Nov 01 19:57:50.094071 ip-10-0-137-175 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 19:57:50.493514 ip-10-0-137-175 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:57:50.493514 ip-10-0-137-175 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 19:57:50.493514 ip-10-0-137-175 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:57:50.493514 ip-10-0-137-175 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:57:50.498249 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498165    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.498249 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498248    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.498249 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498253    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498256    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498259    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498262    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498265    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498268    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498270    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498273    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498276    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498279    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498281    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498284    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498288    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498293    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498296    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498299    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498302    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498304    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498306    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498309    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.498347 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498312    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498315    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498317    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498320    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498322    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498325    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498334    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498336    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498339    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498341    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498344    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498347    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498350    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498353    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498356    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498360    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498365    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498368    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498371    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.498834 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498373    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498376    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498379    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498382    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498384    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498387    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498389    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498392    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498394    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498397    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498399    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498402    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498404    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498407    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498409    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498413    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498416    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498419    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498422    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498424    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.499276 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498430    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498433    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498435    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498438    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498440    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498443    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498445    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498448    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498450    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498452    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498455    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498457    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498460    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498462    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498465    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498467    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498470    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498472    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498475    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498478    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.499743 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498480    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498483    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498485    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498487    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498490    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498492    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498495    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498499    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498502    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498504    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498507    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498509    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498513    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498518    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498521    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498523    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498526    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498528    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498531    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.500252 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498533    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498536    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498538    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498541    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498543    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.498546    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499376    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499383    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499387    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499390    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499394    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499398    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499402    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499405    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499408    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499411    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499414    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499416    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499419    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499421    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.500718 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499427    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499430    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499432    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499435    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499437    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499440    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499443    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499446    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499449    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499451    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499454    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499457    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499460    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499465    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499467    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499470    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499473    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.501221 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499475    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499480    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499484    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499486    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499489    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499492    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499495    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499498    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499500    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499503    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499506    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499509    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499511    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499514    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499518    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499521    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499523    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499526    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499528    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499531    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.501659 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499534    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499536    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499539    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499542    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499545    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499548    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499550    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499553    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499556    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499558    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499561    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499563    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499566    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499568    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499571    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499573    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499576    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499578    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499581    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499584    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.502157 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499586    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499589    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499592    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499594    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499597    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499599    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499604    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499606    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499609    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499612    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499614    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499617    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499619    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499622    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499625    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499628    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499630    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499633    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499635    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.502628 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499638    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499640    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499643    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499645    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499648    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499650    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499653    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499655    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499658    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499660    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499663    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499666    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499669    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499671    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.499675    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500251    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500257    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500261    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500264    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500269    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.503203 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500272    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500274    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500277    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500280    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500282    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500285    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500287    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500290    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500293    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500296    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500300    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500304    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500307    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500310    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500312    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500315    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500317    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500320    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500322    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.504012 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500325    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500328    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500332    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500335    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500339    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500342    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500345    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500347    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500350    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500352    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500355    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500357    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500361    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500364    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500366    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500369    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500372    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500374    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500377    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.504535 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500379    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500381    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500384    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500387    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500389    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500392    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500395    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500397    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500400    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500402    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500405    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500407    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500410    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500412    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500415    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500417    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500419    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500422    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500425    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500428    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.505038 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500430    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500433    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500435    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500438    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500440    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500444    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500446    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500449    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500453    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500455    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500458    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500460    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500463    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500465    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500468    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500470    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500474    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500476    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500479    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500481    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.505515 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500484    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500486    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500489    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500491    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500494    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500496    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500499    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500501    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500504    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500507    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500510    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500513    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500515    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500518    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500521    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500523    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500525    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500529    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500532    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.505996 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500535    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500537    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500541    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.500543    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502444    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502454    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502463    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502468    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502473    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502476    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502481    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502486    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502490    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502493    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502496    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502499    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502503    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502506    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502509    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502512    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502515    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502518    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502521    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502524    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 19:57:50.506471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502527    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502530    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502534    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502538    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502541    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502544    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502548    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502551    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502554    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502557    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502560    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502563    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502567    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502570    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502574    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502577    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502580    2582 flags.go:64] FLAG: --enable-server="true"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502583    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502587    2582 flags.go:64] FLAG: --event-burst="100"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502591    2582 flags.go:64] FLAG: --event-qps="50"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502594    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502598    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502601    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502605    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 19:57:50.507082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502608    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502611    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502614    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502617    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502620    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502624    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502627    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502630    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502633    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502636    2582 flags.go:64] FLAG: --feature-gates=""
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502640    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502643    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502647    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502650    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502654    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502657    2582 flags.go:64] FLAG: --help="false"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502660    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502663    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502666    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502669    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502672    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502676    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502679    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 19:57:50.507654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502682    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502685    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502688    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502691    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502694    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502698    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502701    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502703    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502707    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502710    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502713    2582 flags.go:64] FLAG: --lock-file=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502716    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502719    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502722    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502728    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502730    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502733    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502736    2582 flags.go:64] FLAG: --logging-format="text"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502739    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502742    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502745    2582 flags.go:64] FLAG: --manifest-url=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502748    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502753    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502757    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502761    2582 flags.go:64] FLAG: --max-pods="110"
Nov 01 19:57:50.508206 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502764    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502767    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502770    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502773    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502786    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502789    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502792    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502800    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502803    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502807    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502810    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502813    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502815    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502819    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502822    2582 flags.go:64] FLAG: --port="10250"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502825    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502828    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06b4c5b941d71eb7d"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502831    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502833    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502836    2582 flags.go:64] FLAG: --register-node="true"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502839    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502843    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502846    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502849    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502852    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502856    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 19:57:50.508803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502859    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502862    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502865    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502868    2582 flags.go:64] FLAG: --runonce="false"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502874    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502878    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502881    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502884    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502887    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502890    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502893    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502896    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502899    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502902    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502905    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502908    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502911    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502914    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502917    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.502999    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503659    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503692    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503703    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503709    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503715    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 19:57:50.509438 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503721    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503727    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503733    2582 flags.go:64] FLAG: --v="2"
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503748    2582 flags.go:64] FLAG: --version="false"
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503755    2582 flags.go:64] FLAG: --vmodule=""
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503799    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.503807    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504384    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504397    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504400    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504403    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504406    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504409    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504412    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504414    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504417    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504419    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504422    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504427    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504432    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504435    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.510087 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504438    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504441    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504447    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504450    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504452    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504455    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504458    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504460    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504464    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504467    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504470    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504472    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504475    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504477    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504480    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504482    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504485    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504488    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504490    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504493    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.510584 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504496    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504498    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504501    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504503    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504506    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504508    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504511    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504514    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504516    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504519    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504521    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504524    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504526    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504528    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504533    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504537    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504539    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504542    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504545    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.511425 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504548    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504551    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504553    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504556    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504559    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504561    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504564    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504567    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504569    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504572    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504574    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504577    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504579    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504582    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504584    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504587    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504590    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504592    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504594    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504597    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.512338 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504600    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504602    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504604    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504607    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504609    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504612    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504614    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504619    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504622    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504625    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504627    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504630    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504633    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504636    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504638    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504641    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504643    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504646    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504648    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.513195 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504651    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504653    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504656    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504659    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504662    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504664    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504667    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504669    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504672    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504676    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504680    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504683    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504686    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504689    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.504718    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:57:50.514026 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504761    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504765    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504768    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504771    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504790    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504796    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504801    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504804    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504806    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504809    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504811    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504814    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504817    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504819    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504822    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504824    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504827    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504830    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504832    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.514682 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504835    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504839    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504843    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504845    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504848    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504851    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504853    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504856    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504858    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504861    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504864    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504867    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504869    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504872    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504875    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504877    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504880    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504882    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504886    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504890    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.515541 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504893    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504895    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504898    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504900    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504903    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504905    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504908    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504911    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504913    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504916    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504919    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504922    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504924    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504927    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504930    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504932    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504935    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504937    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504940    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.516036 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504942    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504945    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504948    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504950    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504953    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504956    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504958    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504961    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504964    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504966    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504969    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504973    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504976    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504979    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504981    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504984    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504987    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504990    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504992    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504995    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.516755 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.504997    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505000    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505003    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505005    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505008    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505010    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505013    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505015    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505018    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505020    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505023    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505025    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505028    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505030    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505033    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505035    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505038    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505041    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505044    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.517649 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505047    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505049    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505052    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505054    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505058    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505062    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505064    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505067    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.505070    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.513170    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.513188    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.513210    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.513218    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513306    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513314    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513319    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513327    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513334    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513340    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.518175 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513345    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513349    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513354    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513358    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513362    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513367    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513371    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513376    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513380    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513384    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513389    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513394    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513399    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513403    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513407    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513412    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513416    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513421    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513433    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.518767 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513438    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513442    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513447    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513452    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513457    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513467    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513471    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513475    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513480    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513484    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513488    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513492    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513496    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513500    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513505    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513509    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513513    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513518    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513522    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.519244 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513527    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513531    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513536    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513540    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513568    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513574    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513579    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513585    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513592    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513597    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513602    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513607    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513611    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513616    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513621    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513625    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513629    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513635    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513640    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.519738 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513644    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513648    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513653    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513657    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513661    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513666    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513670    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513674    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513679    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513683    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513687    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513691    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513695    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513700    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513704    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513708    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513714    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513718    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513722    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513727    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.520304 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513731    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513735    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513740    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513745    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513749    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513753    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513758    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513762    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513767    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513771    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513790    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513796    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513801    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513805    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513810    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513814    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513818    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513823    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513827    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513831    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.520835 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513835    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.513840    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.513883    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514145    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514155    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514160    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514165    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514169    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514174    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514179    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514183    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514188    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514192    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514197    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514202    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514206    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.521315 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514211    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514215    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514221    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514229    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514234    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514238    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514242    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514249    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514253    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514258    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514262    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514267    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514271    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514276    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514280    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514284    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514289    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514293    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514297    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.521823 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514301    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514306    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514310    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514314    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514319    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514323    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514328    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514332    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514336    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514340    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514344    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514349    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514353    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514357    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514361    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514366    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514370    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514374    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514378    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514383    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.522284 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514387    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514392    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514397    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514401    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514405    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514409    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514414    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514418    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514422    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514426    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514431    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514435    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514440    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514444    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514448    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514452    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514456    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514460    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514465    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.522763 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514469    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514474    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514478    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514483    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514487    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514491    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514495    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514502    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514508    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514513    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514519    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514526    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514532    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514536    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514541    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514547    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514551    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514555    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.523229 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514559    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514564    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514568    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514573    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514577    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514581    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514586    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514590    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514595    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514599    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514603    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514607    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514612    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514616    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514621    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514625    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.523871 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.514666    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514721    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514729    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514733    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514737    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514742    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514746    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514750    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514754    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514758    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514763    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514768    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514772    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514792    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514797    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514801    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514805    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514809    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514813    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514817    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:57:50.524501 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514821    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514825    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514828    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514833    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514837    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514840    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514844    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514848    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514852    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514856    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514860    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514865    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514869    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514874    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514878    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514882    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514886    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514893    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514900    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:57:50.525375 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514905    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514913    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514919    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514925    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514930    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514934    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514939    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514943    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514948    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514953    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514957    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514962    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514966    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514970    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514974    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514979    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514983    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514988    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514992    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:57:50.526056 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.514996    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515001    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515005    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515010    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515014    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515018    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515023    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515027    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515031    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515036    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515040    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515044    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515048    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515053    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515057    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515060    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515064    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515069    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515073    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:57:50.526531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515081    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515085    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515089    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515094    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515098    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515103    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515107    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515111    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515117    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515122    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515126    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515130    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515135    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515139    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515143    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515147    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515151    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515158    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515162    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515167    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:57:50.527057 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515171    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515175    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515180    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515184    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515188    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515192    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515198    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515202    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515207    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:50.515211    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.515302    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.518490    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.519700    2582 server.go:1014] "Starting client certificate rotation"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.519801    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.519836    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.524076    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:57:50.527531 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.525808    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:57:50.538366 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.538338    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 01 19:57:50.543053 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.543037    2582 log.go:25] "Validated CRI v1 image API"
Nov 01 19:57:50.543864 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.543839    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 19:57:50.549402 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.549380    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b0333523-96f3-4b6c-bbfc-b656ffaeacc3:/dev/nvme0n1p4 dbf1dad0-27e6-4e89-9bd4-dae8b3ca0711:/dev/nvme0n1p3]
Nov 01 19:57:50.549484 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.549405    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 19:57:50.556142 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.556125    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 19:57:50.556194 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.556019    2582 manager.go:217] Machine: {Timestamp:2026-11-01 19:57:50.554462339 +0000 UTC m=+0.352524791 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110464 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28bd46c15cb0b55257f8f00f37dc2f SystemUUID:ec28bd46-c15c-b0b5-5257-f8f00f37dc2f BootID:15d6de52-f155-4dd1-9656-f638a82e6068 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:ef:95:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:ef:95:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:99:4b:40:46:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 19:57:50.556194 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.556157    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 19:57:50.556276 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.556241    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 19:57:50.559098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559071    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 19:57:50.559229 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559099    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-175.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 19:57:50.559275 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559236    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 19:57:50.559275 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559244    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 19:57:50.559275 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559263    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:57:50.559351 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.559278    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:57:50.560386 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.560376    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 19:57:50.564631 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564619    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 19:57:50.564743 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564734    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 19:57:50.564771 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564766    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 19:57:50.564811 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564791    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 19:57:50.564811 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564801    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 19:57:50.564811 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564811    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 01 19:57:50.564913 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.564819    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 19:57:50.565180 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.565169    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:57:50.567721 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.567706    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 19:57:50.574456 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.574434    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:57:50.575388 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575373    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 19:57:50.575463 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575404    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 19:57:50.575572 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575561    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 19:57:50.575617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575577    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 19:57:50.575617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575583    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 19:57:50.575617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575589    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 19:57:50.575617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575595    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 19:57:50.575617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.575601    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 19:57:50.576352 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576342    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 19:57:50.576382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576353    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 19:57:50.576382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576362    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 19:57:50.576382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576368    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 19:57:50.576476 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576387    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 19:57:50.576476 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.576397    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 19:57:50.577274 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.577264    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 19:57:50.577309 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.577276    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 19:57:50.581255 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.581243    2582 server.go:1263] "Started kubelet"
Nov 01 19:57:50.581414 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.581332    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 19:57:50.581414 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.581382    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 19:57:50.581523 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.581441    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 19:57:50.582093 ip-10-0-137-175 systemd[1]: Started Kubernetes Kubelet.
Nov 01 19:57:50.583971 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.583040    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 19:57:50.584163 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.584148    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 19:57:50.589591 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.589536    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 19:57:50.589591 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.589543    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-175.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 19:57:50.589793 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.589614    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-175.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 19:57:50.589793 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.589699    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 19:57:50.592793 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.592768    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 19:57:50.592989 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.591836    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-175.us-east-2.compute.internal.18e4041c284091b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-175.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-175.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 19:57:50.581215665 +0000 UTC m=+0.379278116,LastTimestamp:2026-11-01 19:57:50.581215665 +0000 UTC m=+0.379278116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-175.us-east-2.compute.internal,}"
Nov 01 19:57:50.593473 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.593454    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 19:57:50.593588 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.593569    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:50.594239 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594222    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 19:57:50.594239 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594225    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 19:57:50.594343 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594248    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 19:57:50.594343 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594338    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 19:57:50.594343 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594344    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 19:57:50.594421 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594401    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 19:57:50.594421 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594414    2582 factory.go:55] Registering systemd factory
Nov 01 19:57:50.594493 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594423    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 01 19:57:50.594640 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594626    2582 factory.go:153] Registering CRI-O factory
Nov 01 19:57:50.594680 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594642    2582 factory.go:223] Registration of the crio container factory successfully
Nov 01 19:57:50.594680 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594657    2582 factory.go:103] Registering Raw factory
Nov 01 19:57:50.594680 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.594676    2582 manager.go:1199] Started watching for new ooms in manager
Nov 01 19:57:50.595723 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.595709    2582 manager.go:319] Starting recovery of all containers
Nov 01 19:57:50.599761 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.599729    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-175.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 19:57:50.601251 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.601220    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnbhd"
Nov 01 19:57:50.605855 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.605835    2582 manager.go:324] Recovery completed
Nov 01 19:57:50.606544 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.606519    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 19:57:50.608720 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.608705    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xnbhd"
Nov 01 19:57:50.610928 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.610916    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.612692 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.612674    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.612801 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.612709    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.612801 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.612724    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.613347 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.613334    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 01 19:57:50.613347 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.613346    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 19:57:50.613491 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.613363    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 19:57:50.613491 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.613333    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.614226 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.614208    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.614298 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.614246    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.614298 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.614261    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.614943 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.614873    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-175.us-east-2.compute.internal.18e4041c2a20d980  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-175.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-175.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-175.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 19:57:50.612691328 +0000 UTC m=+0.410753791,LastTimestamp:2026-11-01 19:57:50.612691328 +0000 UTC m=+0.410753791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-175.us-east-2.compute.internal,}"
Nov 01 19:57:50.615350 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.615339    2582 policy_none.go:50] "Start"
Nov 01 19:57:50.615406 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.615356    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 19:57:50.615406 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.615372    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 19:57:50.617338 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.617326    2582 policy_none.go:44] "Start"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.653297    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.655720    2582 manager.go:341] "Starting Device Plugin manager"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.655752    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.655764    2582 server.go:87] "Starting device plugin registration server"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.656022    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.656032    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.656167    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.656236    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.656244    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.656951    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.657004    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.660639    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.661858    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.662531    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.662561    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.662592    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 19:57:50.667628 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.665409    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:57:50.756509 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.756324    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.757472 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.757450    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.757579 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.757491    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.757579 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.757506    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.757579 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.757536    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.762788 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.762758    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"]
Nov 01 19:57:50.762871 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.762831    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.763685 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.763668    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.763800 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.763697    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.763800 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.763707    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.765970 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.765957    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.766122 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.766210 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766137    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.766655 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766635    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.766733 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766669    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.766733 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766641    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.766733 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766700    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.766733 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766679    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.766733 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.766717    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.767184 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.767169    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.767239 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.767194    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-175.us-east-2.compute.internal\": node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:50.768870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.768855    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.768952 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.768886    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:57:50.769582 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.769570    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:57:50.769668 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.769602    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:57:50.769668 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.769612    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:57:50.785946 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.785918    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:50.790227 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.790205    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.792190 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.792175    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.795524 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.795503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.795604 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.795530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.795604 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.795550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.886787 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.886742    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:50.896079 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.896157 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896129    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.896157 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.896235 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.896235 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896188    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.896235 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:50.896198    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:50.987258 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:50.987219    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.088024 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.087963    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.095224 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.095205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:51.097359 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.097345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:51.188523 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.188490    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.289027 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.288978    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.389280 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.389206    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.496406 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.496365    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.519583 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.519553    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 19:57:51.519725 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.519707    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 19:57:51.596498 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.596464    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.615020 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.614960    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:52:51 +0000 UTC" deadline="2028-06-23 13:55:40.005132216 +0000 UTC"
Nov 01 19:57:51.615020 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.615009    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14393h57m48.390126023s"
Nov 01 19:57:51.679669 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.679595    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:57:51.696987 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:51.696962    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-175.us-east-2.compute.internal\" not found"
Nov 01 19:57:51.769868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.769830    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:57:51.793976 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.793943    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:51.803143 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.803121    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:57:51.804370 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.804350    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:57:51.816398 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.816378    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:57:51.932194 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:51.932129    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bb0f42cc122a3cebecd048bef257a8.slice/crio-66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6 WatchSource:0}: Error finding container 66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6: Status 404 returned error can't find the container with id 66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6
Nov 01 19:57:51.932583 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.932563    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:57:51.932766 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:51.932748    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aa65d6c5643e7fc09a137b81d3ea6b.slice/crio-e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6 WatchSource:0}: Error finding container e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6: Status 404 returned error can't find the container with id e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6
Nov 01 19:57:51.939535 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:51.939518    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 19:57:52.568683 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.568643    2582 apiserver.go:52] "Watching apiserver"
Nov 01 19:57:52.578749 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.577415    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 19:57:52.578749 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.578204    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-q4k9b","openshift-ovn-kubernetes/ovnkube-node-789kh","kube-system/global-pull-secret-syncer-d549b","kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-drpqc","openshift-multus/multus-additional-cni-plugins-5d5kw","kube-system/konnectivity-agent-qdn6f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244","openshift-image-registry/node-ca-szb2b","openshift-multus/multus-zzrgf","openshift-multus/network-metrics-daemon-244s9","openshift-dns/node-resolver-q6p64","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-j5t7b"]
Nov 01 19:57:52.582489 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.582465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.584971 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.584952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.585080 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.585020    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:57:52.586727 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.586702    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.586727 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.586720    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\""
Nov 01 19:57:52.586906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.586723    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.586906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.586771    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 19:57:52.586906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.586783    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 19:57:52.587335 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.587321    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.587412 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.587376    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:57:52.590353 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.590329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:52.590439 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.590401    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:57:52.592884 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.592865    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.593165 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.593146    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 19:57:52.595204 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.595184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.597482 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.597464    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.599876 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.599853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.602407 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.602388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.605050 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.605323 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.605419 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.605419 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.605542 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.605542 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.605542 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.605662 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.605662 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.605662 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.605779 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.605779 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgtp\" (UniqueName: \"kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.605779 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.605779 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606002 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606002 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.606002 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606002 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5lx\" (UniqueName: \"kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606002 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.605963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5hj\" (UniqueName: \"kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2s7r\" (UniqueName: \"kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.606748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.607640 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.607640 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.606796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.607933 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.607916    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.610527 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.610507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.614012 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.613941    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 19:57:52.622267 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.622248    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9jjwv"
Nov 01 19:57:52.627660 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.627638    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9jjwv"
Nov 01 19:57:52.668560 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.668505    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerStarted","Data":"66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6"}
Nov 01 19:57:52.670451 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.670422    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"02aa65d6c5643e7fc09a137b81d3ea6b","Type":"ContainerStarted","Data":"e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6"}
Nov 01 19:57:52.695173 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695138    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 19:57:52.695314 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695260    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.695483 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695462    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 19:57:52.695560 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695540    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.695647 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695631    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.695647 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695633    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 19:57:52.695748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695690    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 19:57:52.695748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695729    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.695748 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695737    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.695913 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695807    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 19:57:52.695913 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695833    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\""
Nov 01 19:57:52.695913 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695849    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.695913 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.695865    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\""
Nov 01 19:57:52.696845 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696807    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 19:57:52.696845 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696845    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\""
Nov 01 19:57:52.696845 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696862    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 19:57:52.697074 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696826    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.697074 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696905    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 19:57:52.697074 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.696938    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.697207 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697184    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 19:57:52.697207 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697185    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 19:57:52.697364 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697344    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\""
Nov 01 19:57:52.697446 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697375    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.697446 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697388    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\""
Nov 01 19:57:52.697446 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.697438    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.698204 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.698185    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 19:57:52.698316 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.698296    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\""
Nov 01 19:57:52.698443 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.698398    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\""
Nov 01 19:57:52.699085 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.699065    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 19:57:52.699179 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.699137    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 19:57:52.699179 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.699153    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\""
Nov 01 19:57:52.699304 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.699280    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 19:57:52.708081 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.708185 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708185 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708302 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708220    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708302 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708260    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708444 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.708444 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.708444 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.708444 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bkf\" (UniqueName: \"kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.708713 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.708713 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:52.708713 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2s7r\" (UniqueName: \"kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.708900 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708900 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.708900 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709060 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.709060 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708979    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709060 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.708989    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709060 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709084    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709176    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdz7p\" (UniqueName: \"kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709150    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709456    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709512    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709635    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709807    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5lx\" (UniqueName: \"kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.709902 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709952    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.709991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710008    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5hj\" (UniqueName: \"kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710224    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710296    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.710982 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710475    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710588    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfpz\" (UniqueName: \"kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710762    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.710783    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:52.711781 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710891    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.710945    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:53.210920525 +0000 UTC m=+3.008982969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.710988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdb9\" (UniqueName: \"kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711142    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711152    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711280    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwn8\" (UniqueName: \"kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.712741 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgtp\" (UniqueName: \"kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.711891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.712236    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.712313    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.712364    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:57:53.212347733 +0000 UTC m=+3.010410176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.712418    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.713699 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.712491    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.712749    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.713007    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.713012    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.713071    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.713079    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.713079    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.714150    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.714174    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.714196    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:52.714546 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:52.714270    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:53.214242116 +0000 UTC m=+3.012304582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:52.715124 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.714966    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.715829 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.715783    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.717362 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.717312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2s7r\" (UniqueName: \"kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.718139 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.718113    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5hj\" (UniqueName: \"kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.721679 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.721657    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5lx\" (UniqueName: \"kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.723085 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.723049    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgtp\" (UniqueName: \"kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:52.812523 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812523 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.812523 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdb9\" (UniqueName: \"kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812611    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwn8\" (UniqueName: \"kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812705    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812764    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.812934 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812893    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.812961    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813065    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bkf\" (UniqueName: \"kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813193    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdz7p\" (UniqueName: \"kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.813470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813445    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813486    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813544    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813579    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813629    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813646    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813665    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813716    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.814345 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813742    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.813981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814087    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814240    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814295    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfpz\" (UniqueName: \"kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814350    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.815880 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814404    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.815880 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814651    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815880 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814695    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.815880 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.814869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.818082 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.818046    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.818537 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.818507    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.827233 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.827180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdz7p\" (UniqueName: \"kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.827682 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.827660    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bkf\" (UniqueName: \"kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.828439 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.828402    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdb9\" (UniqueName: \"kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.829638 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.829610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwn8\" (UniqueName: \"kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.830804 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.830777    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfpz\" (UniqueName: \"kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.896056 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.896010    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzrgf"
Nov 01 19:57:52.905053 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.905025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:57:52.913142 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.913101    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f68e3cb_db93_4919_acb4_b60652805f67.slice/crio-3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007 WatchSource:0}: Error finding container 3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007: Status 404 returned error can't find the container with id 3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007
Nov 01 19:57:52.917076 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.917031    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc020f3c7_45a8_4226_9e41_289a4a7ab737.slice/crio-4852dc1ba944c4f141d4fc49a9a78d57aaf663409c41efc07e65255b2ed86f13 WatchSource:0}: Error finding container 4852dc1ba944c4f141d4fc49a9a78d57aaf663409c41efc07e65255b2ed86f13: Status 404 returned error can't find the container with id 4852dc1ba944c4f141d4fc49a9a78d57aaf663409c41efc07e65255b2ed86f13
Nov 01 19:57:52.922976 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.922952    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:57:52.927915 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.927883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:57:52.929991 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.929971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:57:52.935376 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.935351    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:57:52.940277 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.940244    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d00d01_ecba_4d49_aa5c_c9ab0e6e6867.slice/crio-c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d WatchSource:0}: Error finding container c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d: Status 404 returned error can't find the container with id c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d
Nov 01 19:57:52.943990 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.943960    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a1313c_bd53_44ed_8984_6fc0f890fc3c.slice/crio-4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855 WatchSource:0}: Error finding container 4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855: Status 404 returned error can't find the container with id 4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855
Nov 01 19:57:52.947529 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.947501    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5534dc_85a6_4a5f_8101_7fc41ef4999e.slice/crio-bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3 WatchSource:0}: Error finding container bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3: Status 404 returned error can't find the container with id bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3
Nov 01 19:57:52.950552 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.950531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:57:52.958288 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.958262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:57:52.961139 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.961057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:57:52.963218 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.963193    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb64076_bc8a_41d9_86b7_179dba150c84.slice/crio-3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246 WatchSource:0}: Error finding container 3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246: Status 404 returned error can't find the container with id 3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246
Nov 01 19:57:52.963551 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:52.963535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:57:52.971914 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:57:52.971885    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d887126_1d78_4f44_afaa_28b9d682e211.slice/crio-53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b WatchSource:0}: Error finding container 53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b: Status 404 returned error can't find the container with id 53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b
Nov 01 19:57:53.217357 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.217269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:53.217357 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.217315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:53.217593 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217445    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:57:53.217593 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217457    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:53.217593 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.217457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:53.217593 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217526    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:53.217593 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217463    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:57:53.217867 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217609    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:53.217867 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217531    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:57:54.217513084 +0000 UTC m=+4.015575542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:53.217867 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217675    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:54.217657791 +0000 UTC m=+4.015720237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:53.217867 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:53.217709    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:54.217695771 +0000 UTC m=+4.015758212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:53.631889 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.631545    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:52:53 +0000 UTC" deadline="2028-06-10 05:12:07.051625341 +0000 UTC"
Nov 01 19:57:53.631889 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.631617    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14073h14m13.420013371s"
Nov 01 19:57:53.681122 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.681083    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerStarted","Data":"4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855"}
Nov 01 19:57:53.685472 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.685419    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szb2b" event={"ID":"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867","Type":"ContainerStarted","Data":"c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d"}
Nov 01 19:57:53.687676 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.687576    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drpqc" event={"ID":"c020f3c7-45a8-4226-9e41-289a4a7ab737","Type":"ContainerStarted","Data":"4852dc1ba944c4f141d4fc49a9a78d57aaf663409c41efc07e65255b2ed86f13"}
Nov 01 19:57:53.691223 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.691193    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzrgf" event={"ID":"7f68e3cb-db93-4919-acb4-b60652805f67","Type":"ContainerStarted","Data":"3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007"}
Nov 01 19:57:53.694215 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.694188    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6p64" event={"ID":"9666c411-ce1d-4592-82de-51b88b02ea47","Type":"ContainerStarted","Data":"8e1173f67c1aad4a20e4c989e6cb7ebff89ce0dbc710219213037e462ac6d300"}
Nov 01 19:57:53.695842 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.695814    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qdn6f" event={"ID":"a1c41cc7-e221-4bc2-aec2-af48980dc5eb","Type":"ContainerStarted","Data":"835afd727aad8d22650bfdc2c425d18a0ada2db54591acce6b4ff99d59955a48"}
Nov 01 19:57:53.705388 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.702818    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b"}
Nov 01 19:57:53.705585 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.705525    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4k9b" event={"ID":"cbb64076-bc8a-41d9-86b7-179dba150c84","Type":"ContainerStarted","Data":"3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246"}
Nov 01 19:57:53.710471 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:53.709302    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244" event={"ID":"ab5534dc-85a6-4a5f-8101-7fc41ef4999e","Type":"ContainerStarted","Data":"bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3"}
Nov 01 19:57:54.230429 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.230388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:54.230638 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.230453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:54.230638 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.230508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:54.230763 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.230682    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:57:54.230763 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.230706    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:57:54.230763 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.230719    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:54.230932 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.230785    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:56.230764255 +0000 UTC m=+6.028826745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:54.231417 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.231238    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:54.231417 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.231293    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:57:56.23127722 +0000 UTC m=+6.029339666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:54.231417 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.231352    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:54.231417 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.231386    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:57:56.23137488 +0000 UTC m=+6.029437329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:54.632383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.632271    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:52:53 +0000 UTC" deadline="2028-05-16 04:33:19.752261194 +0000 UTC"
Nov 01 19:57:54.632383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.632329    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13472h35m25.119937748s"
Nov 01 19:57:54.666149 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.666070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:54.666149 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.666107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:54.666369 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.666291    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:57:54.666433 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:54.666371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:54.667714 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.666503    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:57:54.667714 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:54.666592    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:57:55.764550 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:55.763497    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"02aa65d6c5643e7fc09a137b81d3ea6b","Type":"ContainerStarted","Data":"b968fb6a70d7a12d9bff9e54eec47a62c99ee7afb2b4fa4244dc6452f5c2527c"}
Nov 01 19:57:55.779905 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:55.778695    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5bb0f42cc122a3cebecd048bef257a8" containerID="c448c93da915add7c3239b421da195658cebc36dd393beb1b2bba26a70c0f1c4" exitCode=0
Nov 01 19:57:55.779905 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:55.778744    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerDied","Data":"c448c93da915add7c3239b421da195658cebc36dd393beb1b2bba26a70c0f1c4"}
Nov 01 19:57:55.784847 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:55.784292    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal" podStartSLOduration=3.784276698 podStartE2EDuration="3.784276698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:57:55.779610604 +0000 UTC m=+5.577673070" watchObservedRunningTime="2026-11-01 19:57:55.784276698 +0000 UTC m=+5.582339165"
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.257290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.257364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.257402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.257539    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.257604    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:58:00.257585798 +0000 UTC m=+10.055648243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.257988    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.258022    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:57:56.258035 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.258036    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:56.258482 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.258085    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:00.258070571 +0000 UTC m=+10.056133027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:57:56.258482 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.257990    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:56.258482 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.258165    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:00.258140608 +0000 UTC m=+10.056203078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.664413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.664562    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.664659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.664783    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.664871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:56.665220 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:56.664974    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:57:56.785326 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.785282    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerStarted","Data":"6927e1ece8f53e3036193fcda0b3f5861586c53aaf28c20be614fa1e3c4d6d31"}
Nov 01 19:57:56.797578 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.797420    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="146fd66eec32c00574298c9e34fe43dcd2faab3dfce09c2dd5de875b7dc41ffc" exitCode=0
Nov 01 19:57:56.798625 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.798521    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"146fd66eec32c00574298c9e34fe43dcd2faab3dfce09c2dd5de875b7dc41ffc"}
Nov 01 19:57:56.803328 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:56.802692    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" podStartSLOduration=4.802674445 podStartE2EDuration="4.802674445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:57:56.801777322 +0000 UTC m=+6.599839789" watchObservedRunningTime="2026-11-01 19:57:56.802674445 +0000 UTC m=+6.600736914"
Nov 01 19:57:57.803267 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:57.803230    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qdn6f" event={"ID":"a1c41cc7-e221-4bc2-aec2-af48980dc5eb","Type":"ContainerStarted","Data":"1825bba2c293028b7f5cee799e644bb49c7122ac55db193bad8c10704a746c7f"}
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:58.665888    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:58.666045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:58.666097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:58.666191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:58.666245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:57:58.666436 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:57:58.666339    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:57:59.809442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:59.809305    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244" event={"ID":"ab5534dc-85a6-4a5f-8101-7fc41ef4999e","Type":"ContainerStarted","Data":"314b019eefebaf9135cf0a98b4be922b0c03800c27826ea1ffa41c28b84c56c3"}
Nov 01 19:57:59.812524 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:59.811754    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szb2b" event={"ID":"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867","Type":"ContainerStarted","Data":"2f9a222f0670e570a39944958f1fbd90c3ca56d93fed86914ecf9e358d7c4a3c"}
Nov 01 19:57:59.829517 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:57:59.828350    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qdn6f" podStartSLOduration=4.607574855 podStartE2EDuration="8.828331295s" totalImagesPullingTime="4.22075644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:57:57.820373606 +0000 UTC m=+7.618436072" watchObservedRunningTime="2026-11-01 19:57:59.828331295 +0000 UTC m=+9.626393757"
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.290083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.290192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.290253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.290405    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.290429    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.290441    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.290500    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.290482621 +0000 UTC m=+18.088545065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.291153    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.291238    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.291212258 +0000 UTC m=+18.089274696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.291288    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:00.291382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.291316    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:58:08.291307596 +0000 UTC m=+18.089370035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:00.669603 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.669527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:00.669756 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.669683    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:00.669756 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.669747    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:00.672988 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.672961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:00.673151 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.672984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:00.673151 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:00.673078    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:00.994848 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.994144    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:00.995922 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:00.995899    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:01.011129 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.011074    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-szb2b" podStartSLOduration=4.162683078 podStartE2EDuration="10.011053081s" totalImagesPullingTime="5.848370003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:57:59.829223224 +0000 UTC m=+9.627285685" watchObservedRunningTime="2026-11-01 19:58:01.011053081 +0000 UTC m=+10.809115543"
Nov 01 19:58:01.541906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.541870    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 19:58:01.670700 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.670611    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:58:01.541890055Z","UUID":"75a095e6-7954-4e8c-8454-41494d5df10f","Handler":null,"Name":"","Endpoint":""}
Nov 01 19:58:01.675020 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.674603    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 19:58:01.675020 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.674631    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 19:58:01.820742 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.820708    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244" event={"ID":"ab5534dc-85a6-4a5f-8101-7fc41ef4999e","Type":"ContainerStarted","Data":"6f926b78b0f1fe0ead4c1749021c5b480023d87f2556cf9f08ee6c6fd0225641"}
Nov 01 19:58:01.823054 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.823022    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerStarted","Data":"949bdb4c0cd3867cbf0aff4e165826ea4c60d8b165726a91b23990e582ac403a"}
Nov 01 19:58:01.824705 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.824678    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzrgf" event={"ID":"7f68e3cb-db93-4919-acb4-b60652805f67","Type":"ContainerStarted","Data":"19263d89ed06aac618dc9ac9ecdf299def64513edde29741e79eb9d12e612bf3"}
Nov 01 19:58:01.826862 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.826833    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q6p64" event={"ID":"9666c411-ce1d-4592-82de-51b88b02ea47","Type":"ContainerStarted","Data":"8c9781be09058658680439469bff720b452c8622be4ba40e2b24930f65647ba9"}
Nov 01 19:58:01.859363 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.859310    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q6p64" podStartSLOduration=2.418034663 podStartE2EDuration="10.859289677s" totalImagesPullingTime="8.441255014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:01.859053303 +0000 UTC m=+11.657115765" watchObservedRunningTime="2026-11-01 19:58:01.859289677 +0000 UTC m=+11.657352139"
Nov 01 19:58:01.875377 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:01.875328    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zzrgf" podStartSLOduration=2.590698887 podStartE2EDuration="10.875314192s" totalImagesPullingTime="8.284615305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:01.874846914 +0000 UTC m=+11.672909370" watchObservedRunningTime="2026-11-01 19:58:01.875314192 +0000 UTC m=+11.673376694"
Nov 01 19:58:02.663162 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:02.662903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:02.663600 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:02.662903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:02.663600 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:02.663218    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:02.663600 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:02.662903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:02.663600 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:02.663284    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:02.663600 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:02.663352    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:02.830658 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:02.830620    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="949bdb4c0cd3867cbf0aff4e165826ea4c60d8b165726a91b23990e582ac403a" exitCode=0
Nov 01 19:58:02.831718 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:02.831690    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"949bdb4c0cd3867cbf0aff4e165826ea4c60d8b165726a91b23990e582ac403a"}
Nov 01 19:58:03.833990 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:03.833955    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4k9b" event={"ID":"cbb64076-bc8a-41d9-86b7-179dba150c84","Type":"ContainerStarted","Data":"11e137cda8b0b62c30d49cde2aefed44c8f918be47cc45ea761e8bc7a780f20a"}
Nov 01 19:58:03.836207 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:03.836178    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244" event={"ID":"ab5534dc-85a6-4a5f-8101-7fc41ef4999e","Type":"ContainerStarted","Data":"de118c796a104436c2948b71a460564a94f8866b6e31b6a6e5e2166d57a8bd9d"}
Nov 01 19:58:03.853990 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:03.853947    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q4k9b" podStartSLOduration=4.238405674 podStartE2EDuration="12.853934699s" totalImagesPullingTime="8.615529025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:03.853493817 +0000 UTC m=+13.651556279" watchObservedRunningTime="2026-11-01 19:58:03.853934699 +0000 UTC m=+13.651997160"
Nov 01 19:58:04.663829 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.663794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:04.664006 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.663802    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:04.664006 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.663924    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:04.664006 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:04.663922    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:04.664124 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:04.664019    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:04.664124 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:04.664098    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:04.839377 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.839338    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drpqc" event={"ID":"c020f3c7-45a8-4226-9e41-289a4a7ab737","Type":"ContainerStarted","Data":"af948cc4d708c95880893178a579568bb15c1a25142dd7a20f4814306febfbcb"}
Nov 01 19:58:04.841943 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.841916    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="900c2ddc585fe6568905524e64b281e9489c402c07ac4eb1fb22b734ce4ecdda" exitCode=0
Nov 01 19:58:04.842061 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.841999    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"900c2ddc585fe6568905524e64b281e9489c402c07ac4eb1fb22b734ce4ecdda"}
Nov 01 19:58:04.856346 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.856300    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-drpqc" podStartSLOduration=2.883979699 podStartE2EDuration="13.856288231s" totalImagesPullingTime="10.972308532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:04.85579519 +0000 UTC m=+14.653857648" watchObservedRunningTime="2026-11-01 19:58:04.856288231 +0000 UTC m=+14.654350691"
Nov 01 19:58:04.856555 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:04.856530    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244" podStartSLOduration=4.015264461 podStartE2EDuration="13.856524769s" totalImagesPullingTime="9.841260308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:03.875714204 +0000 UTC m=+13.673776667" watchObservedRunningTime="2026-11-01 19:58:04.856524769 +0000 UTC m=+14.654587290"
Nov 01 19:58:05.846573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:05.846394    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="c07a2593be736e9924acc4267d4a162e9ac2672237f5947b87c91900b41c6c42" exitCode=0
Nov 01 19:58:05.847081 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:05.846482    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"c07a2593be736e9924acc4267d4a162e9ac2672237f5947b87c91900b41c6c42"}
Nov 01 19:58:06.638010 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.637976    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:06.638174 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.638116    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:06.639078 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.638980    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:06.662961 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.662938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:06.663109 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:06.663047    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:06.663362 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.663339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:06.663474 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:06.663440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:06.663518 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:06.663490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:06.663555 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:06.663533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:08.348320 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.348295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.348341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.348384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348427    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348477    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret podName:acc92340-bb89-4ff1-838e-08a6574531b9 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:24.348463355 +0000 UTC m=+34.146525810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret") pod "global-pull-secret-syncer-d549b" (UID: "acc92340-bb89-4ff1-838e-08a6574531b9") : object "kube-system"/"original-pull-secret" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348499    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348517    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348533    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348546    2582 projected.go:196] Error preparing data for projected volume kube-api-access-bj5s2 for pod openshift-network-diagnostics/network-check-target-j5t7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348548    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:58:24.348535706 +0000 UTC m=+34.146598149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 19:58:08.348697 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.348603    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2 podName:efd428db-ed11-4811-b94d-76b9e12e4288 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:24.34858942 +0000 UTC m=+34.146651859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5s2" (UniqueName: "kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2") pod "network-check-target-j5t7b" (UID: "efd428db-ed11-4811-b94d-76b9e12e4288") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 19:58:08.662946 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.662861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:08.662946 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.662883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:08.663193 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.662992    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:08.663193 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.663041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:08.663193 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.663124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:08.663193 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:08.663185    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:08.855340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855308    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"deda08e15ae057f076fe6b248490a657415ebbcca591f12a7a34fa07f0e45af7"}
Nov 01 19:58:08.855486 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855374    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"e86246f12f9441c3b0942627dbdbac5428986f7e61991ef7481cfa3dbfd9aac4"}
Nov 01 19:58:08.855486 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855417    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"df64348df419fb8ff19b80d95886f0897f18d8b4cb6dc9c3814e221c035f96af"}
Nov 01 19:58:08.855486 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855447    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"4c0bff5c71fcc2a93709b904b66db0fd9d1d529dc93633c1d7cc104d3a3b350b"}
Nov 01 19:58:08.855486 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855474    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"65a2aa7cc00e67143d651745812d0b0300303c7e2baea3d9b813a380257b327b"}
Nov 01 19:58:08.855640 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.855506    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"67e45b4505994d32e8f6a2ca445adfc21059241496c6ff9096be96c134e6efc0"}
Nov 01 19:58:08.858058 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.858036    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="6f0ecb62906b93669dde4a64dde4242e6b30a32145c1ac3c6fac10e363656199" exitCode=0
Nov 01 19:58:08.858127 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:08.858074    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"6f0ecb62906b93669dde4a64dde4242e6b30a32145c1ac3c6fac10e363656199"}
Nov 01 19:58:09.863257 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:09.863225    2582 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="f8ff7a56188eb127004f99916300e77097f629f78f85f29e561ec8ceb683fc22" exitCode=0
Nov 01 19:58:09.863257 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:09.863261    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerDied","Data":"f8ff7a56188eb127004f99916300e77097f629f78f85f29e561ec8ceb683fc22"}
Nov 01 19:58:10.663593 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.663562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:10.663742 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.663671    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:10.663742 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:10.663675    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:10.663742 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.663712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:10.663894 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:10.663801    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:10.663945 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:10.663896    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:10.868321 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.868241    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"44c7dda84fcf2c3ca334f21cdeda6cb64cd3d2991ecfa8458ba5b3a6180463c2"}
Nov 01 19:58:10.871336 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.871317    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" event={"ID":"25a1313c-bd53-44ed-8984-6fc0f890fc3c","Type":"ContainerStarted","Data":"f1c4f2d8a79b14c79bbdda69705375e7ebdff99fc5fc0feb63998d6daf896441"}
Nov 01 19:58:10.893112 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:10.893062    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5d5kw" podStartSLOduration=7.258772419 podStartE2EDuration="19.89304976s" totalImagesPullingTime="12.634277341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:10.892649374 +0000 UTC m=+20.690711835" watchObservedRunningTime="2026-11-01 19:58:10.89304976 +0000 UTC m=+20.691112221"
Nov 01 19:58:12.663573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.663552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:12.664130 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.663579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:12.664130 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:12.663660    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:12.664130 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.663716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:12.664130 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:12.663819    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:12.664130 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:12.663849    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:12.878149 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.877899    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" event={"ID":"7d887126-1d78-4f44-afaa-28b9d682e211","Type":"ContainerStarted","Data":"784034ac28a1768e862e26251e9a0e473f9d1fde29d93a06adde0f427a8943b5"}
Nov 01 19:58:12.878149 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.878114    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:12.893717 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.893694    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:12.905291 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:12.905252    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-789kh" podStartSLOduration=6.774332145 podStartE2EDuration="21.905240713s" totalImagesPullingTime="15.130908568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:12.904244183 +0000 UTC m=+22.702306644" watchObservedRunningTime="2026-11-01 19:58:12.905240713 +0000 UTC m=+22.703303173"
Nov 01 19:58:13.301538 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:13.301462    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:13.881484 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:13.881393    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:13.898204 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:13.898173    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:14.342010 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.341924    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d549b"]
Nov 01 19:58:14.342227 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.342080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:14.342227 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:14.342165    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:14.345268 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.345238    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5t7b"]
Nov 01 19:58:14.345408 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.345362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:14.345477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:14.345430    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:14.345820 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.345798    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-244s9"]
Nov 01 19:58:14.345915 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:14.345886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:14.345996 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:14.345974    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:15.663374 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:15.663346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:15.663753 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:15.663444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:16.662810 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:16.662765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:16.663008 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:16.662912    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:16.663008 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:16.662957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:16.663125 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:16.663069    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:17.663639 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:17.663557    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:17.664074 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:17.663660    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:18.663012 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:18.662929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:18.663012 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:18.662950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:18.663228 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:18.663058    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-244s9" podUID="8961d680-2f89-4358-a9fc-ff43d01e2eda"
Nov 01 19:58:18.663228 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:18.663172    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5t7b" podUID="efd428db-ed11-4811-b94d-76b9e12e4288"
Nov 01 19:58:19.663399 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:19.663374    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:19.663683 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:19.663474    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d549b" podUID="acc92340-bb89-4ff1-838e-08a6574531b9"
Nov 01 19:58:20.057853 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.057765    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeReady"
Nov 01 19:58:20.057981 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.057896    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 19:58:20.091174 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.091143    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ppbp7"]
Nov 01 19:58:20.095040 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.095020    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8n2gw"]
Nov 01 19:58:20.095172 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.095159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:20.099570 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.099026    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kdsg"]
Nov 01 19:58:20.099570 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.099110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:20.102552 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.102535    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:58:20.102674 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.102658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:20.106074 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.106054    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"]
Nov 01 19:58:20.106289 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.106272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.107573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.107553    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.107648 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.107611    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.110553 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.110534    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"]
Nov 01 19:58:20.110727 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.110706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.114031 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.114012    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-846f8d77f5-km598"]
Nov 01 19:58:20.114140 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.114120    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.117087 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.117073    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"]
Nov 01 19:58:20.117200 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.117186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.120331 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.120315    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:58:20.120420 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.120406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.124837 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.123563    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"]
Nov 01 19:58:20.124837 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.123811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.129240 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129219    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 19:58:20.129324 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129308    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j2hms\""
Nov 01 19:58:20.129378 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129354    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 19:58:20.129430 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129401    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 19:58:20.129480 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129472    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 19:58:20.129536 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129519    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 19:58:20.129588 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.129570    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 19:58:20.130612 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.130591    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 19:58:20.131636 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.131619    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"]
Nov 01 19:58:20.131850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.131695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:20.136875 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.136856    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"]
Nov 01 19:58:20.137242 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.137202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.140481 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.140460    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4rbdb"]
Nov 01 19:58:20.140655 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.140548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.143759 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.143742    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"]
Nov 01 19:58:20.143875 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.143841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.146868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.146849    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"]
Nov 01 19:58:20.149232 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.149214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.151004 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.150369    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6svs9"]
Nov 01 19:58:20.151371 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.151204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153584    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"]
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153618    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"]
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153643    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"]
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153668    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153690    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"]
Nov 01 19:58:20.154263 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.153711    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-74zbl"]
Nov 01 19:58:20.155216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.154816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.158172 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158151    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kdsg"]
Nov 01 19:58:20.158251 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158184    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8n2gw"]
Nov 01 19:58:20.158251 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158211    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ppbp7"]
Nov 01 19:58:20.158251 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158229    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"]
Nov 01 19:58:20.158251 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158242    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:58:20.158383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158255    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4rbdb"]
Nov 01 19:58:20.158383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158272    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"]
Nov 01 19:58:20.158383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158297    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"]
Nov 01 19:58:20.158383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158315    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6svs9"]
Nov 01 19:58:20.158383 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.158566 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158388    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"]
Nov 01 19:58:20.158566 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158407    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-846f8d77f5-km598"]
Nov 01 19:58:20.158566 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.158475    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74zbl"]
Nov 01 19:58:20.208853 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.208830    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 19:58:20.209298 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209250    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.209298 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209289    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209327    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209327    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209332    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209398    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209405    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209414    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209419    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209433    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 19:58:20.209448 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209452    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.209870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209463    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.209870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209471    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 19:58:20.209870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209465    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.209870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.209503    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.211184 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.211168    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.211269 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.211223    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.212761 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.212736    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.212870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.212770    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.212870 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.212842    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.214293 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.214275    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.214381 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.214276    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.214747 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.214731    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.214844 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.214765    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.217275 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.217261    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 19:58:20.218195 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.218180    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 19:58:20.218511 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.218497    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:20.220632 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.220618    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 19:58:20.230513 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.230496    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 19:58:20.232125 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232106    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232123    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232160    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232166    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232181    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232191    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 19:58:20.232221 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232199    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232247    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232276    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232303    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n856f\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232305    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232314    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\""
Nov 01 19:58:20.232441 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.232361    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\""
Nov 01 19:58:20.235507 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.235550 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f4b\" (UniqueName: \"kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.235550 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.235649 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.235649 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.235747 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx67m\" (UniqueName: \"kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.235747 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.235747 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.235850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.235850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.235850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljsg\" (UniqueName: \"kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.235850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbfs\" (UniqueName: \"kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs\") pod \"network-check-source-6f78d59f99-djwlz\" (UID: \"2fad2018-8963-46b5-a399-e165e48dde59\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:20.235968 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.235968 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.236036 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.235979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z45gb\" (UniqueName: \"kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.236036 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.236036 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.236145 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66grf\" (UniqueName: \"kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.236145 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.236145 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.236145 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dg6\" (UniqueName: \"kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.236280 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.236280 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.236280 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.236280 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.236470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236314    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wlz\" (UniqueName: \"kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.236470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.236470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.236470 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.236633 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bgd\" (UniqueName: \"kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd\") pod \"kas-connection-checker-685895c7c9-5kdsg\" (UID: \"6fb413b1-e547-411a-a27a-50716d2bd1ef\") " pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:20.236633 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.236633 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.236633 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.236633 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkc5\" (UniqueName: \"kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5\") pod \"kas-connection-checker-685895c7c9-8n2gw\" (UID: \"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4\") " pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.236868 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzbc\" (UniqueName: \"kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc\") pod \"volume-data-source-validator-58c9f5f46d-qrm2q\" (UID: \"2b1aad29-2c51-448a-92cd-abc01fe696d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.236973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvt6\" (UniqueName: \"kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.237098 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxg5c\" (UniqueName: \"kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjb2f\" (UniqueName: \"kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f\") pod \"kas-connection-checker-685895c7c9-ppbp7\" (UID: \"087c078b-50e9-4942-ac02-6faf11ffecf3\") " pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.237340 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.237260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.338377 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.338377 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljsg\" (UniqueName: \"kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbfs\" (UniqueName: \"kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs\") pod \"network-check-source-6f78d59f99-djwlz\" (UID: \"2fad2018-8963-46b5-a399-e165e48dde59\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z45gb\" (UniqueName: \"kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.338576 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66grf\" (UniqueName: \"kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dg6\" (UniqueName: \"kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338714    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wlz\" (UniqueName: \"kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.338897 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338912    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bgd\" (UniqueName: \"kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd\") pod \"kas-connection-checker-685895c7c9-5kdsg\" (UID: \"6fb413b1-e547-411a-a27a-50716d2bd1ef\") " pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.338991    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkc5\" (UniqueName: \"kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5\") pod \"kas-connection-checker-685895c7c9-8n2gw\" (UID: \"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4\") " pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.339162    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.839141527 +0000 UTC m=+30.637203982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:20.339442 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339368    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339496    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.339563    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.338990    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.339622    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.839606625 +0000 UTC m=+30.637669071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : secret "router-metrics-certs-default" not found
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.339766    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.339880    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:20.340054 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.339928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.839912264 +0000 UTC m=+30.637974721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzbc\" (UniqueName: \"kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc\") pod \"volume-data-source-validator-58c9f5f46d-qrm2q\" (UID: \"2b1aad29-2c51-448a-92cd-abc01fe696d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340641    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvt6\" (UniqueName: \"kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg5c\" (UniqueName: \"kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340968    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.341573 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.340998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb2f\" (UniqueName: \"kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f\") pod \"kas-connection-checker-685895c7c9-ppbp7\" (UID: \"087c078b-50e9-4942-ac02-6faf11ffecf3\") " pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f4b\" (UniqueName: \"kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341253    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.841239431 +0000 UTC m=+30.639301888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341284    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341301    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b96df89b-pmb2t: secret "image-registry-tls" not found
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341334    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341356    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls podName:60703b7e-ae49-4bec-923c-73ade0101e57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.841340291 +0000 UTC m=+30.639402735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls") pod "image-registry-59b96df89b-pmb2t" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57") : secret "image-registry-tls" not found
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.341373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls podName:6533ed5f-4009-45be-8d6f-b90f1848d640 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.841364613 +0000 UTC m=+30.639427063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls") pod "dns-default-6svs9" (UID: "6533ed5f-4009-45be-8d6f-b90f1848d640") : secret "dns-default-metrics-tls" not found
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341593    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.342477 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341756    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.344156 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.343960    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.344156 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.344078    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.344302 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.344186    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:20.344302 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.344201    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77f5b9cf45-wqbcb: secret "image-registry-tls" not found
Nov 01 19:58:20.344302 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.344248    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls podName:c8da3aea-f54a-45ad-b9bf-4a8108f7951e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.844234013 +0000 UTC m=+30.642296460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls") pod "image-registry-77f5b9cf45-wqbcb" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e") : secret "image-registry-tls" not found
Nov 01 19:58:20.344667 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.344637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.344667 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341209    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.344667 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.344656    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.344888 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.344765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.344963 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.344942    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.345139 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345115    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.345283 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345263    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.345329 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345205    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345719    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.346104    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.346162    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:20.846140128 +0000 UTC m=+30.644202609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.341283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.346339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.346382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx67m\" (UniqueName: \"kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.346413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.346455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.346009 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.346486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.348313 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.348286    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.349510 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.349168    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.349593 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.349547    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.350611 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.350054    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.350611 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.350035    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dg6\" (UniqueName: \"kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.350611 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.345932    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.350611 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.350414    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljsg\" (UniqueName: \"kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.350863 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.350837    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.351421 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.351151    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.353093 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.353058    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkc5\" (UniqueName: \"kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5\") pod \"kas-connection-checker-685895c7c9-8n2gw\" (UID: \"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4\") " pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:20.353385 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.353361    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.354803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.353959    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbfs\" (UniqueName: \"kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs\") pod \"network-check-source-6f78d59f99-djwlz\" (UID: \"2fad2018-8963-46b5-a399-e165e48dde59\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:20.354803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.353978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.354803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.354314    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66grf\" (UniqueName: \"kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.354803 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.354589    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.355300 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.355261    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bgd\" (UniqueName: \"kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd\") pod \"kas-connection-checker-685895c7c9-5kdsg\" (UID: \"6fb413b1-e547-411a-a27a-50716d2bd1ef\") " pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:20.355725 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.355699    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z45gb\" (UniqueName: \"kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.357303 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.357271    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f4b\" (UniqueName: \"kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.358292 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.358269    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvt6\" (UniqueName: \"kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.358511 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.358473    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wlz\" (UniqueName: \"kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.359540 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.359379    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.359540 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.359510    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg5c\" (UniqueName: \"kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.359721 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.359695    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb2f\" (UniqueName: \"kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f\") pod \"kas-connection-checker-685895c7c9-ppbp7\" (UID: \"087c078b-50e9-4942-ac02-6faf11ffecf3\") " pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:20.359997 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.359972    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx67m\" (UniqueName: \"kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.360344 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.360324    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.360987 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.360969    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzbc\" (UniqueName: \"kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc\") pod \"volume-data-source-validator-58c9f5f46d-qrm2q\" (UID: \"2b1aad29-2c51-448a-92cd-abc01fe696d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:20.410013 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.409990    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:20.416426 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.416407    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:20.421759 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.421680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:20.472410 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.472365    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:20.483964 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.483939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:20.489494 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.489311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:20.495810 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.495315    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:20.502402 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.502229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:20.514261 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.514057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:20.525620 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.525443    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:20.664240 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.664211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:20.664240 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.664236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:20.668000 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.667973    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 19:58:20.668000 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.667977    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\""
Nov 01 19:58:20.668130 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.668061    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\""
Nov 01 19:58:20.708852 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.708830    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8n2gw"]
Nov 01 19:58:20.712968 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.712941    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5kdsg"]
Nov 01 19:58:20.713208 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.713180    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e2c9d1_9ec9_4aa6_baed_78ca5facc4f4.slice/crio-e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c WatchSource:0}: Error finding container e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c: Status 404 returned error can't find the container with id e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c
Nov 01 19:58:20.714284 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.714263    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ppbp7"]
Nov 01 19:58:20.715220 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.715199    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087c078b_50e9_4942_ac02_6faf11ffecf3.slice/crio-2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07 WatchSource:0}: Error finding container 2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07: Status 404 returned error can't find the container with id 2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07
Nov 01 19:58:20.716416 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.716392    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb413b1_e547_411a_a27a_50716d2bd1ef.slice/crio-50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887 WatchSource:0}: Error finding container 50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887: Status 404 returned error can't find the container with id 50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887
Nov 01 19:58:20.783187 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.782943    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"]
Nov 01 19:58:20.786461 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.786439    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"]
Nov 01 19:58:20.789654 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.789619    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053d72d0_c2a0_4e08_a7bf_4a6c1c7174d0.slice/crio-4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f WatchSource:0}: Error finding container 4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f: Status 404 returned error can't find the container with id 4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f
Nov 01 19:58:20.798019 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.797980    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c95e40_824d_468a_99f4_57ec8b88fb42.slice/crio-b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640 WatchSource:0}: Error finding container b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640: Status 404 returned error can't find the container with id b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640
Nov 01 19:58:20.822495 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.822474    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"]
Nov 01 19:58:20.825939 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.825916    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4rbdb"]
Nov 01 19:58:20.828759 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.828731    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1aad29_2c51_448a_92cd_abc01fe696d1.slice/crio-34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587 WatchSource:0}: Error finding container 34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587: Status 404 returned error can't find the container with id 34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587
Nov 01 19:58:20.829023 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.829000    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfbd1ed_3a02_4334_b370_328f2a511729.slice/crio-4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243 WatchSource:0}: Error finding container 4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243: Status 404 returned error can't find the container with id 4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243
Nov 01 19:58:20.845284 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.845222    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"]
Nov 01 19:58:20.848336 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.848316    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"]
Nov 01 19:58:20.850726 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.849728    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74zbl"]
Nov 01 19:58:20.850726 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.850573    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fad2018_8963_46b5_a399_e165e48dde59.slice/crio-1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614 WatchSource:0}: Error finding container 1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614: Status 404 returned error can't find the container with id 1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614
Nov 01 19:58:20.851844 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:20.851822    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b456599_bf71_4bab_afa7_e2978c6e4e42.slice/crio-fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60 WatchSource:0}: Error finding container fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60: Status 404 returned error can't find the container with id fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60
Nov 01 19:58:20.853271 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.853271 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:20.853271 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:20.853271 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:20.853271 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:20.853504 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:20.853504 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:20.853504 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.853423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:20.853745 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853656    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:20.853745 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853675    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77f5b9cf45-wqbcb: secret "image-registry-tls" not found
Nov 01 19:58:20.853745 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853730    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls podName:c8da3aea-f54a-45ad-b9bf-4a8108f7951e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.853710703 +0000 UTC m=+31.651773147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls") pod "image-registry-77f5b9cf45-wqbcb" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e") : secret "image-registry-tls" not found
Nov 01 19:58:20.853975 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853841    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.853827802 +0000 UTC m=+31.651890254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:20.853975 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853903    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:20.853975 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853940    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b96df89b-pmb2t: secret "image-registry-tls" not found
Nov 01 19:58:20.854120 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.853989    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls podName:60703b7e-ae49-4bec-923c-73ade0101e57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.853978061 +0000 UTC m=+31.652040512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls") pod "image-registry-59b96df89b-pmb2t" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57") : secret "image-registry-tls" not found
Nov 01 19:58:20.854120 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854064    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:20.854120 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854097    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.854086721 +0000 UTC m=+31.652149166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:20.854316 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854151    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:20.854316 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854205    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.854183354 +0000 UTC m=+31.652245796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:20.854316 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854263    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 19:58:20.854316 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854301    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls podName:6533ed5f-4009-45be-8d6f-b90f1848d640 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.854287265 +0000 UTC m=+31.652349706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls") pod "dns-default-6svs9" (UID: "6533ed5f-4009-45be-8d6f-b90f1848d640") : secret "dns-default-metrics-tls" not found
Nov 01 19:58:20.854494 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854398    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 19:58:20.854494 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854423    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.854416793 +0000 UTC m=+31.652479237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : secret "router-metrics-certs-default" not found
Nov 01 19:58:20.854835 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854815    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:20.854884 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:20.854873    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:21.854856621 +0000 UTC m=+31.652919075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:20.895989 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.895962    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q" event={"ID":"2b1aad29-2c51-448a-92cd-abc01fe696d1","Type":"ContainerStarted","Data":"34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587"}
Nov 01 19:58:20.897147 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.897112    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" event={"ID":"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0","Type":"ContainerStarted","Data":"4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f"}
Nov 01 19:58:20.902291 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.899054    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8n2gw" event={"ID":"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4","Type":"ContainerStarted","Data":"2263faede792019eea03d34e0695721b40265c47ae242763f244011e3bb177ae"}
Nov 01 19:58:20.902291 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.899110    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8n2gw" event={"ID":"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4","Type":"ContainerStarted","Data":"e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c"}
Nov 01 19:58:20.903932 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.903903    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj" event={"ID":"9b456599-bf71-4bab-afa7-e2978c6e4e42","Type":"ContainerStarted","Data":"fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60"}
Nov 01 19:58:20.905260 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.905239    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5kdsg" event={"ID":"6fb413b1-e547-411a-a27a-50716d2bd1ef","Type":"ContainerStarted","Data":"ff512ff2bd4ed04ea23a9e35f87aed01594e8954ec85fb963737def481e93093"}
Nov 01 19:58:20.905352 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.905278    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5kdsg" event={"ID":"6fb413b1-e547-411a-a27a-50716d2bd1ef","Type":"ContainerStarted","Data":"50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887"}
Nov 01 19:58:20.906363 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.906341    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74zbl" event={"ID":"a5aab77d-3a73-46b8-ba03-03e96ebf16f0","Type":"ContainerStarted","Data":"f41f87af22f6fdc0c5c32873ac96cdc9fe52c00391ca816e183ace2a4977f8f2"}
Nov 01 19:58:20.907355 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.907335    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz" event={"ID":"2fad2018-8963-46b5-a399-e165e48dde59","Type":"ContainerStarted","Data":"1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614"}
Nov 01 19:58:20.908629 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.908603    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ppbp7" event={"ID":"087c078b-50e9-4942-ac02-6faf11ffecf3","Type":"ContainerStarted","Data":"f043fc4a7114b5340f0943b1783bfa113ff4b80d2f42eb41b6dbf57c2d8219cd"}
Nov 01 19:58:20.908693 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.908632    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ppbp7" event={"ID":"087c078b-50e9-4942-ac02-6faf11ffecf3","Type":"ContainerStarted","Data":"2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07"}
Nov 01 19:58:20.909724 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.909701    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg" event={"ID":"46c95e40-824d-468a-99f4-57ec8b88fb42","Type":"ContainerStarted","Data":"b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640"}
Nov 01 19:58:20.910589 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.910572    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerStarted","Data":"4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243"}
Nov 01 19:58:20.920513 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.920478    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8n2gw" podStartSLOduration=436.920467614 podStartE2EDuration="7m16.920467614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:20.919860512 +0000 UTC m=+30.717922971" watchObservedRunningTime="2026-11-01 19:58:20.920467614 +0000 UTC m=+30.718530075"
Nov 01 19:58:20.943877 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.943839    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5kdsg" podStartSLOduration=436.943827797 podStartE2EDuration="7m16.943827797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:20.943438785 +0000 UTC m=+30.741501248" watchObservedRunningTime="2026-11-01 19:58:20.943827797 +0000 UTC m=+30.741890250"
Nov 01 19:58:20.960599 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:20.960564    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ppbp7" podStartSLOduration=436.960552886 podStartE2EDuration="7m16.960552886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:20.959654379 +0000 UTC m=+30.757716846" watchObservedRunningTime="2026-11-01 19:58:20.960552886 +0000 UTC m=+30.758615347"
Nov 01 19:58:21.665151 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.665116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:21.669254 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.669207    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\""
Nov 01 19:58:21.669390 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.669271    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:21.867985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868122    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868138    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b96df89b-pmb2t: secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868201    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls podName:60703b7e-ae49-4bec-923c-73ade0101e57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.868176425 +0000 UTC m=+33.666238878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls") pod "image-registry-59b96df89b-pmb2t" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57") : secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868586    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868637    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.86862241 +0000 UTC m=+33.666684864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868695    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868706    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77f5b9cf45-wqbcb: secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls podName:c8da3aea-f54a-45ad-b9bf-4a8108f7951e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.86873109 +0000 UTC m=+33.666793543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls") pod "image-registry-77f5b9cf45-wqbcb" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e") : secret "image-registry-tls" not found
Nov 01 19:58:21.869216 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868813    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868844    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls podName:6533ed5f-4009-45be-8d6f-b90f1848d640 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.868835094 +0000 UTC m=+33.666897548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls") pod "dns-default-6svs9" (UID: "6533ed5f-4009-45be-8d6f-b90f1848d640") : secret "dns-default-metrics-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868899    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868927    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.868918477 +0000 UTC m=+33.666980934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.868992    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.869019    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.869010095 +0000 UTC m=+33.667072548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : secret "router-metrics-certs-default" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.869069    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.869098    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.86908894 +0000 UTC m=+33.667151393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:21.871747 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:21.869158    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:23.869150023 +0000 UTC m=+33.667212475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:23.894498 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.894419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:23.894498 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.894486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894540    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894615    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.89459462 +0000 UTC m=+37.692657077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894635    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894642    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894662    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77f5b9cf45-wqbcb: secret "image-registry-tls" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.894530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894719    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls podName:c8da3aea-f54a-45ad-b9bf-4a8108f7951e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.894694576 +0000 UTC m=+37.692757030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls") pod "image-registry-77f5b9cf45-wqbcb" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e") : secret "image-registry-tls" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.894749    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls podName:6533ed5f-4009-45be-8d6f-b90f1848d640 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.894733869 +0000 UTC m=+37.692796324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls") pod "dns-default-6svs9" (UID: "6533ed5f-4009-45be-8d6f-b90f1848d640") : secret "dns-default-metrics-tls" not found
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.894867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.894962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:23.895014 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.895004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895060    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895112    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.895097917 +0000 UTC m=+37.693160373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.895063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895142    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.895129192 +0000 UTC m=+37.693191638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895180    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895215    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.895203803 +0000 UTC m=+37.693266256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : secret "router-metrics-certs-default" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.895237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895281    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895318    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.895307018 +0000 UTC m=+37.693369464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895344    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895356    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b96df89b-pmb2t: secret "image-registry-tls" not found
Nov 01 19:58:23.895521 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:23.895388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls podName:60703b7e-ae49-4bec-923c-73ade0101e57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.895377255 +0000 UTC m=+37.693439724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls") pod "image-registry-59b96df89b-pmb2t" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57") : secret "image-registry-tls" not found
Nov 01 19:58:23.937405 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.937372    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q" event={"ID":"2b1aad29-2c51-448a-92cd-abc01fe696d1","Type":"ContainerStarted","Data":"56575096d90d2cef67454f8d29168e693f70beaee9e53cbbcf6654719a7aaa4b"}
Nov 01 19:58:23.953989 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:23.952690    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q" podStartSLOduration=410.219619517 podStartE2EDuration="6m52.952671567s" totalImagesPullingTime="2.73305205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:23.952019424 +0000 UTC m=+33.750081886" watchObservedRunningTime="2026-11-01 19:58:23.952671567 +0000 UTC m=+33.750734029"
Nov 01 19:58:24.399819 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.399742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:24.400117 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.399936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:24.400117 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.399993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:24.400628 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:24.400605    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 19:58:24.400740 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:24.400682    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs podName:8961d680-2f89-4358-a9fc-ff43d01e2eda nodeName:}" failed. No retries permitted until 2026-11-01 19:58:56.400664409 +0000 UTC m=+66.198726866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs") pod "network-metrics-daemon-244s9" (UID: "8961d680-2f89-4358-a9fc-ff43d01e2eda") : secret "metrics-daemon-secret" not found
Nov 01 19:58:24.405638 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.405612    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:24.405758 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.405741    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:24.583101 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.583067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:24.694882 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.694751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:24.889016 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.888709    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5t7b"]
Nov 01 19:58:24.943921 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:24.943849    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5t7b" event={"ID":"efd428db-ed11-4811-b94d-76b9e12e4288","Type":"ContainerStarted","Data":"7a23a6ef453d8ce8f60a2f80877a9a759326861058322f770a0c4e59f4483e0d"}
Nov 01 19:58:25.169831 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.169519    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d549b"]
Nov 01 19:58:25.177824 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:25.177741    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc92340_bb89_4ff1_838e_08a6574531b9.slice/crio-5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e WatchSource:0}: Error finding container 5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e: Status 404 returned error can't find the container with id 5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e
Nov 01 19:58:25.949323 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.949293    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d549b" event={"ID":"acc92340-bb89-4ff1-838e-08a6574531b9","Type":"ContainerStarted","Data":"5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e"}
Nov 01 19:58:25.951584 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.951438    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg" event={"ID":"46c95e40-824d-468a-99f4-57ec8b88fb42","Type":"ContainerStarted","Data":"f018a66de2ac8f647473e5032475e974c3ace75265d476e8e99971ed01dace97"}
Nov 01 19:58:25.954556 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.954437    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerStarted","Data":"c0cfd4fd8aced840fbb1071afcf11574419b4effb12a00fccb6737bd7e18953b"}
Nov 01 19:58:25.956804 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.956191    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/0.log"
Nov 01 19:58:25.956804 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.956219    2582 generic.go:356] "Generic (PLEG): container finished" podID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" containerID="27196fdc08bbe87d2a2fe1184d68d28c955ff08f2eef297e433843b0058e297d" exitCode=255
Nov 01 19:58:25.956804 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.956288    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" event={"ID":"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0","Type":"ContainerDied","Data":"27196fdc08bbe87d2a2fe1184d68d28c955ff08f2eef297e433843b0058e297d"}
Nov 01 19:58:25.956804 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.956562    2582 scope.go:122] "RemoveContainer" containerID="27196fdc08bbe87d2a2fe1184d68d28c955ff08f2eef297e433843b0058e297d"
Nov 01 19:58:25.959040 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:25.959001    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj" event={"ID":"9b456599-bf71-4bab-afa7-e2978c6e4e42","Type":"ContainerStarted","Data":"b34162eb7a6cb25d1bb71c96bdec1a8e7994a8b8e9a4ed4bee7ea279e1080848"}
Nov 01 19:58:26.424971 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.422558    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-922ms"]
Nov 01 19:58:26.454100 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.453447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.459283 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.458889    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:58:26.460820 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.459887    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 19:58:26.460820 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.459893    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\""
Nov 01 19:58:26.460820 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.459944    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 19:58:26.488979 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.487899    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-922ms"]
Nov 01 19:58:26.488979 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.487967    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:58:26.488979 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.488075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.523478 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.523446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.523716 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.523698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.523830 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.523816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.524000 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.523986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gr9\" (UniqueName: \"kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.524103 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.524091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.524261 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.524233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.524320 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.524304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.524370 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.524337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.524420 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.524386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.554349 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.553135    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96bd4\""
Nov 01 19:58:26.620341 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.620268    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"]
Nov 01 19:58:26.625134 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.625134 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.625134 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gr9\" (UniqueName: \"kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.625452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:26.626083    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:26.626160    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.126125308 +0000 UTC m=+36.924187761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : configmap "kube-rbac-proxy" not found
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.626933    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:26.627043    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:26.627141    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:27.127121011 +0000 UTC m=+36.925183469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.627659    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.627690 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.627673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.632906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.627705    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.632906 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.628205    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.645039 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.645017    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"]
Nov 01 19:58:26.656814 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.645244    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:26.656814 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.649136    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.656814 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.649431    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 19:58:26.656814 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.650235    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgvh2\""
Nov 01 19:58:26.657439 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.657421    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gr9\" (UniqueName: \"kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:26.687440 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.685078    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"]
Nov 01 19:58:26.687440 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.685115    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"]
Nov 01 19:58:26.687440 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.685240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:26.726950 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.726530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv5d\" (UniqueName: \"kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d\") pod \"migrator-66df5d496b-5b24w\" (UID: \"5bbad68d-9021-4665-9315-d4ac8521f567\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:26.726950 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.726694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75gk\" (UniqueName: \"kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk\") pod \"migrator-66df5d496b-gdpfq\" (UID: \"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:26.750469 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.749868    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:26.828033 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.828005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75gk\" (UniqueName: \"kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk\") pod \"migrator-66df5d496b-gdpfq\" (UID: \"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:26.828522 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.828182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv5d\" (UniqueName: \"kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d\") pod \"migrator-66df5d496b-5b24w\" (UID: \"5bbad68d-9021-4665-9315-d4ac8521f567\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:26.836559 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.836530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75gk\" (UniqueName: \"kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk\") pod \"migrator-66df5d496b-gdpfq\" (UID: \"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:26.838208 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.838184    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv5d\" (UniqueName: \"kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d\") pod \"migrator-66df5d496b-5b24w\" (UID: \"5bbad68d-9021-4665-9315-d4ac8521f567\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:26.859391 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.859364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:26.968217 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.968182    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74zbl" event={"ID":"a5aab77d-3a73-46b8-ba03-03e96ebf16f0","Type":"ContainerStarted","Data":"83e850cbeba53e6ebd10ae2518b3e94de1c980c03b796096ba9a5cf1bc77dd4d"}
Nov 01 19:58:26.971146 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.971116    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz" event={"ID":"2fad2018-8963-46b5-a399-e165e48dde59","Type":"ContainerStarted","Data":"79471860e593ba059b03f74e35b5975c4525b746720c1f6fca9c5d7d63c12b29"}
Nov 01 19:58:26.973771 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.973750    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:26.974152 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.974136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/0.log"
Nov 01 19:58:26.974220 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.974164    2582 generic.go:356] "Generic (PLEG): container finished" podID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb" exitCode=255
Nov 01 19:58:26.974617 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.974602    2582 scope.go:122] "RemoveContainer" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"
Nov 01 19:58:26.974760 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:26.974742    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pk8qq_openshift-console-operator(053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" podUID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0"
Nov 01 19:58:26.974939 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.974923    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" event={"ID":"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0","Type":"ContainerDied","Data":"cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"}
Nov 01 19:58:26.975005 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.974964    2582 scope.go:122] "RemoveContainer" containerID="27196fdc08bbe87d2a2fe1184d68d28c955ff08f2eef297e433843b0058e297d"
Nov 01 19:58:26.988254 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.986891    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-74zbl" podStartSLOduration=1.4853416400000001 podStartE2EDuration="6.986875658s" totalImagesPullingTime="5.501534018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:26.986152847 +0000 UTC m=+36.784215309" watchObservedRunningTime="2026-11-01 19:58:26.986875658 +0000 UTC m=+36.784938121"
Nov 01 19:58:26.994403 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:26.994044    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:27.020346 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.020213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:27.029959 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.029904    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg" podStartSLOduration=411.0082807 podStartE2EDuration="6m56.029883663s" totalImagesPullingTime="5.021602963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:27.010528268 +0000 UTC m=+36.808590728" watchObservedRunningTime="2026-11-01 19:58:27.029883663 +0000 UTC m=+36.827946126"
Nov 01 19:58:27.049085 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.049033    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz" podStartSLOduration=409.530150611 podStartE2EDuration="6m55.049013393s" totalImagesPullingTime="5.518862782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:27.028924606 +0000 UTC m=+36.826987068" watchObservedRunningTime="2026-11-01 19:58:27.049013393 +0000 UTC m=+36.847075857"
Nov 01 19:58:27.067070 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.066036    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj" podStartSLOduration=411.109838107 podStartE2EDuration="6m56.066018044s" totalImagesPullingTime="4.956179937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:27.049436453 +0000 UTC m=+36.847498914" watchObservedRunningTime="2026-11-01 19:58:27.066018044 +0000 UTC m=+36.864080505"
Nov 01 19:58:27.067671 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.067622    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" podStartSLOduration=411.087854114 podStartE2EDuration="6m56.067602863s" totalImagesPullingTime="4.979748749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:27.065081032 +0000 UTC m=+36.863143494" watchObservedRunningTime="2026-11-01 19:58:27.067602863 +0000 UTC m=+36.865665327"
Nov 01 19:58:27.141273 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.141157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:27.142014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.141587    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 19:58:27.142014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.141660    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:28.141640952 +0000 UTC m=+37.939703406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : configmap "kube-rbac-proxy" not found
Nov 01 19:58:27.142014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.141668    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:27.142014 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.141590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:27.142014 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.141707    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:28.141696656 +0000 UTC m=+37.939759099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:27.159012 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.155761    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:58:27.163846 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:27.163772    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc769b88a_d848_424a_bacb_79d80ea49e48.slice/crio-2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc WatchSource:0}: Error finding container 2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc: Status 404 returned error can't find the container with id 2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc
Nov 01 19:58:27.301493 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.301393    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"]
Nov 01 19:58:27.306145 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:27.306105    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888a6838_2cc9_4bc5_9ca0_52f1f4052ef4.slice/crio-3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00 WatchSource:0}: Error finding container 3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00: Status 404 returned error can't find the container with id 3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00
Nov 01 19:58:27.372092 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.372058    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"]
Nov 01 19:58:27.879037 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.878971    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-62pdr"]
Nov 01 19:58:27.917842 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.917810    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-62pdr"]
Nov 01 19:58:27.917987 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.917905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:27.922607 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.922586    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xrvt7\""
Nov 01 19:58:27.923042 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.923005    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 19:58:27.923042 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.923031    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 19:58:27.923042 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.923041    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 19:58:27.923230 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.923150    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 19:58:27.950455 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:27.950572 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:27.950572 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:27.950572 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:27.950737 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:27.950737 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:27.950737 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:27.950922 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj62\" (UniqueName: \"kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:27.950922 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:27.950922 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:27.950922 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.950873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:27.951117 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951000    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.950986925 +0000 UTC m=+45.749049378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:27.951382 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951363    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:27.951446 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951386    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b96df89b-pmb2t: secret "image-registry-tls" not found
Nov 01 19:58:27.951446 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951430    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls podName:60703b7e-ae49-4bec-923c-73ade0101e57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.951415754 +0000 UTC m=+45.749478207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls") pod "image-registry-59b96df89b-pmb2t" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57") : secret "image-registry-tls" not found
Nov 01 19:58:27.951529 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951490    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:27.951529 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951522    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.951512015 +0000 UTC m=+45.749574468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:27.951613 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951596    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 19:58:27.951613 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951610    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77f5b9cf45-wqbcb: secret "image-registry-tls" not found
Nov 01 19:58:27.951721 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951637    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls podName:c8da3aea-f54a-45ad-b9bf-4a8108f7951e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.951627748 +0000 UTC m=+45.749690199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls") pod "image-registry-77f5b9cf45-wqbcb" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e") : secret "image-registry-tls" not found
Nov 01 19:58:27.951721 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951686    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 19:58:27.951721 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951713    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls podName:6533ed5f-4009-45be-8d6f-b90f1848d640 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.95170313 +0000 UTC m=+45.749765583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls") pod "dns-default-6svs9" (UID: "6533ed5f-4009-45be-8d6f-b90f1848d640") : secret "dns-default-metrics-tls" not found
Nov 01 19:58:27.951892 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951789    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:27.951892 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951825    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.951812742 +0000 UTC m=+45.749875199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:27.951995 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951902    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 19:58:27.951995 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951931    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.951921456 +0000 UTC m=+45.749983908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : secret "router-metrics-certs-default" not found
Nov 01 19:58:27.951995 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.951982    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:27.952121 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.952009    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.952001201 +0000 UTC m=+45.750063655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:27.980593 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.980567    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:27.981019 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.980981    2582 scope.go:122] "RemoveContainer" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"
Nov 01 19:58:27.981221 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:27.981181    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pk8qq_openshift-console-operator(053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" podUID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0"
Nov 01 19:58:27.982466 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.982442    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w" event={"ID":"5bbad68d-9021-4665-9315-d4ac8521f567","Type":"ContainerStarted","Data":"5e156f9f8aa72ad595edd188f5ee5d207b5540ad3bad30b64919cf4a0420555f"}
Nov 01 19:58:27.984088 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.984006    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq" event={"ID":"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4","Type":"ContainerStarted","Data":"3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00"}
Nov 01 19:58:27.987661 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.987639    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t2q78-k5stm" event={"ID":"c769b88a-d848-424a-bacb-79d80ea49e48","Type":"ContainerStarted","Data":"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"}
Nov 01 19:58:27.987762 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:27.987684    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t2q78-k5stm" event={"ID":"c769b88a-d848-424a-bacb-79d80ea49e48","Type":"ContainerStarted","Data":"2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc"}
Nov 01 19:58:28.015486 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.015411    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-t2q78-k5stm" podStartSLOduration=2.015393605 podStartE2EDuration="2.015393605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:28.01473927 +0000 UTC m=+37.812801731" watchObservedRunningTime="2026-11-01 19:58:28.015393605 +0000 UTC m=+37.813456067"
Nov 01 19:58:28.052174 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.052139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.052314 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.052268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj62\" (UniqueName: \"kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.052464 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.052449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.057200 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.054160    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.064654 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.064610    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.073614 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.073586    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj62\" (UniqueName: \"kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.153669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.153822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:28.153995    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:28.154042    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:30.154029896 +0000 UTC m=+39.952092347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:28.154393    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 19:58:28.154519 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:28.154440    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:30.154427002 +0000 UTC m=+39.952489454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : configmap "kube-rbac-proxy" not found
Nov 01 19:58:28.246815 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.246764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:28.549493 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.549469    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-62pdr"]
Nov 01 19:58:28.552165 ip-10-0-137-175 kubenswrapper[2582]: W1101 19:58:28.552129    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544b6ff8_e957_4b48_880f_2311ad4aba68.slice/crio-715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9 WatchSource:0}: Error finding container 715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9: Status 404 returned error can't find the container with id 715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9
Nov 01 19:58:28.994957 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.994873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr" event={"ID":"544b6ff8-e957-4b48-880f-2311ad4aba68","Type":"ContainerStarted","Data":"99567e115ac9d0d9bf0deb84e8cf6e550b00751bcb776b3944edcab5aeaf2570"}
Nov 01 19:58:28.994957 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:28.994929    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr" event={"ID":"544b6ff8-e957-4b48-880f-2311ad4aba68","Type":"ContainerStarted","Data":"715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9"}
Nov 01 19:58:29.018818 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:29.016335    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr" podStartSLOduration=2.016319479 podStartE2EDuration="2.016319479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:29.01602564 +0000 UTC m=+38.814088104" watchObservedRunningTime="2026-11-01 19:58:29.016319479 +0000 UTC m=+38.814381940"
Nov 01 19:58:29.427723 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:29.427691    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6p64_9666c411-ce1d-4592-82de-51b88b02ea47/dns-node-resolver/0.log"
Nov 01 19:58:29.999941 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:29.999906    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5t7b" event={"ID":"efd428db-ed11-4811-b94d-76b9e12e4288","Type":"ContainerStarted","Data":"47ec1fed5a222167a39e326e5dc183f67a9a2da194799f040b176c524e2b42bf"}
Nov 01 19:58:30.000374 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.000029    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:30.001842 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.001814    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w" event={"ID":"5bbad68d-9021-4665-9315-d4ac8521f567","Type":"ContainerStarted","Data":"09195b6376a4542b55ff9d7da288761d21ac869302d881b540d74a47e0f37cb9"}
Nov 01 19:58:30.003478 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.003450    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq" event={"ID":"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4","Type":"ContainerStarted","Data":"004d892de93c70c0e088a3e6a539e5ca74b53af626cc2a1cea53b6da04ff4e81"}
Nov 01 19:58:30.016716 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.016670    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5t7b" podStartSLOduration=34.959673559 podStartE2EDuration="39.016656644s" totalImagesPullingTime="4.056983085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:30.014547267 +0000 UTC m=+39.812609730" watchObservedRunningTime="2026-11-01 19:58:30.016656644 +0000 UTC m=+39.814719136"
Nov 01 19:58:30.032688 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.032637    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq" podStartSLOduration=1.897514948 podStartE2EDuration="4.032623579s" totalImagesPullingTime="2.135108631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:30.030999385 +0000 UTC m=+39.829061844" watchObservedRunningTime="2026-11-01 19:58:30.032623579 +0000 UTC m=+39.830686040"
Nov 01 19:58:30.046257 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.044380    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w" podStartSLOduration=1.993437846 podStartE2EDuration="4.044362947s" totalImagesPullingTime="2.050925101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:30.043990192 +0000 UTC m=+39.842052655" watchObservedRunningTime="2026-11-01 19:58:30.044362947 +0000 UTC m=+39.842425408"
Nov 01 19:58:30.174342 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.174308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:30.174487 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.174409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:30.174487 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:30.174464    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:30.174562 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:30.174529    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 19:58:30.174591 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:30.174532    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.174517751 +0000 UTC m=+43.972580190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:30.174591 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:30.174586    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.174571095 +0000 UTC m=+43.972633541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : configmap "kube-rbac-proxy" not found
Nov 01 19:58:30.472514 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.472477    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:30.472683 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.472616    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:30.472989 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.472974    2582 scope.go:122] "RemoveContainer" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"
Nov 01 19:58:30.473226 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:30.473205    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pk8qq_openshift-console-operator(053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" podUID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0"
Nov 01 19:58:30.628850 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:30.628820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szb2b_f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/node-ca/0.log"
Nov 01 19:58:31.008143 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:31.008116    2582 scope.go:122] "RemoveContainer" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"
Nov 01 19:58:31.008579 ip-10-0-137-175 kubenswrapper[2582]: E1101 19:58:31.008276    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-pk8qq_openshift-console-operator(053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" podUID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0"
Nov 01 19:58:31.027474 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:31.027451    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74zbl_a5aab77d-3a73-46b8-ba03-03e96ebf16f0/serve-healthcheck-canary/0.log"
Nov 01 19:58:31.228895 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:31.228862    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5b24w_5bbad68d-9021-4665-9315-d4ac8521f567/migrator/0.log"
Nov 01 19:58:31.428537 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:31.428486    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-gdpfq_888a6838-2cc9-4bc5-9ca0-52f1f4052ef4/migrator/0.log"
Nov 01 19:58:31.629799 ip-10-0-137-175 kubenswrapper[2582]: I1101 19:58:31.629741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj_9b456599-bf71-4bab-afa7-e2978c6e4e42/kube-storage-version-migrator-operator/0.log"
Nov 01 19:58:32.213201 ip-10-0-137-175 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 19:58:32.222195 ip-10-0-137-175 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 19:58:32.222359 ip-10-0-137-175 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 19:58:32.222530 ip-10-0-137-175 systemd[1]: kubelet.service: Consumed 3.188s CPU time.
Nov 01 19:58:32.237360 ip-10-0-137-175 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 19:58:32.337794 ip-10-0-137-175 kubenswrapper[5041]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:32.337794 ip-10-0-137-175 kubenswrapper[5041]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 19:58:32.337794 ip-10-0-137-175 kubenswrapper[5041]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:32.337794 ip-10-0-137-175 kubenswrapper[5041]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 19:58:32.339858 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339753    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.339858 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339856    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.339858 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339860    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339863    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339866    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339869    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339872    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339875    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339877    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339880    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339883    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339885    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339888    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339891    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339894    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339896    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339899    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339902    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339904    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339907    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339909    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.339966 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339912    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339915    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339917    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339920    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339922    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339930    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339934    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339936    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339940    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339943    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339945    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339948    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339951    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339953    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339956    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339959    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339962    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339965    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339968    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.340447 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339971    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339974    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339976    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339979    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339982    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339985    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339988    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339991    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339993    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.339997    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340000    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340003    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340005    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340008    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340011    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340014    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340016    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340019    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340022    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340025    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.340930 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340027    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340030    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340033    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340035    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340038    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340042    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340044    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340047    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340050    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340052    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340055    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340057    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340060    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340062    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340065    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340067    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340070    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340072    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340075    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340078    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.341518 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340081    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340084    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340086    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340088    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340091    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340093    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340096    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340098    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340101    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340104    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340107    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340112    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340116    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340118    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340121    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340124    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340127    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340130    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340132    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.342019 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340135    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340137    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340140    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340144    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340149    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340153    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340156    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340917    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340925    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340928    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340931    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340934    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340938    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340941    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340944    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340947    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340949    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340952    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340955    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.342492 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340958    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340960    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340964    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340967    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340969    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340972    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340974    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340977    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340980    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340983    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340985    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340988    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340990    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340993    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340996    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.340998    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341001    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341003    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341006    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341009    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.342990 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341011    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341014    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341016    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341019    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341022    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341025    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341028    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341031    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341033    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341036    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341038    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341041    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341044    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341046    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341050    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341055    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341058    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341061    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341064    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.343496 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341066    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341069    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341071    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341074    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341076    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341079    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341081    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341084    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341087    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341090    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341092    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341095    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341098    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341100    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341103    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341105    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341108    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341111    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341113    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.343968 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341116    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341119    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341122    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341125    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341127    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341131    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341133    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341136    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341138    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341141    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341144    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341146    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341149    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341151    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341154    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341157    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341159    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341163    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341166    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.344485 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341169    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341173    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341176    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341179    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341182    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341185    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341188    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341191    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341193    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341196    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341198    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341205    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341207    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341210    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341213    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341215    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341218    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341719    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341724    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341727    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.344958 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341730    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341733    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341737    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341741    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341744    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341747    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341750    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341752    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341755    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341758    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341760    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341763    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341766    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341768    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341771    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341773    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341790    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341793    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341796    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.345454 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341798    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341801    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341804    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341806    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341811    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341814    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341816    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341819    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341822    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341825    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341828    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341830    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341833    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341836    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341838    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341841    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341843    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341846    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341848    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.346065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341851    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341853    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341855    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341858    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341861    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341863    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341866    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341868    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341871    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341874    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341876    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341879    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341881    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341884    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341887    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341889    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341892    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341896    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341899    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341901    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.346856 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341904    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341906    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341909    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341918    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341920    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341923    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341926    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341928    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341931    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341933    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341936    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341940    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341944    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341947    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341950    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341953    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341956    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341959    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341961    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341964    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.347517 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341966    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341969    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341972    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341974    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341977    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341980    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341982    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341985    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341987    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341992    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341995    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.341997    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342000    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342002    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342006    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342008    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342011    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342014    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342017    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342019    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.348061 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342022    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342025    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342027    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342030    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342033    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342122    5041 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342129    5041 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342137    5041 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342144    5041 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342151    5041 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342156    5041 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342162    5041 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342166    5041 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342170    5041 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342173    5041 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342177    5041 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342180    5041 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342183    5041 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342186    5041 flags.go:64] FLAG: --cgroup-root=""
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342189    5041 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342192    5041 flags.go:64] FLAG: --client-ca-file=""
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342195    5041 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342198    5041 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 19:58:32.348536 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342202    5041 flags.go:64] FLAG: --cluster-domain=""
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342204    5041 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342207    5041 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342211    5041 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342214    5041 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342218    5041 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342222    5041 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342225    5041 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342229    5041 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342234    5041 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342238    5041 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342241    5041 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342244    5041 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342247    5041 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342251    5041 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342254    5041 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342257    5041 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342259    5041 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342262    5041 flags.go:64] FLAG: --enable-server="true"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342265    5041 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342270    5041 flags.go:64] FLAG: --event-burst="100"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342273    5041 flags.go:64] FLAG: --event-qps="50"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342276    5041 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342279    5041 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 19:58:32.349100 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342282    5041 flags.go:64] FLAG: --eviction-hard=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342286    5041 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342289    5041 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342292    5041 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342295    5041 flags.go:64] FLAG: --eviction-soft=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342298    5041 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342301    5041 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342304    5041 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342307    5041 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342310    5041 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342313    5041 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342316    5041 flags.go:64] FLAG: --feature-gates=""
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342321    5041 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342324    5041 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342327    5041 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342330    5041 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342334    5041 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342338    5041 flags.go:64] FLAG: --help="false"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342341    5041 flags.go:64] FLAG: --hostname-override="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342344    5041 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342347    5041 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342350    5041 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342354    5041 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 19:58:32.349690 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342357    5041 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342360    5041 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342363    5041 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342366    5041 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342368    5041 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342371    5041 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342374    5041 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342377    5041 flags.go:64] FLAG: --kube-reserved=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342380    5041 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342383    5041 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342386    5041 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342389    5041 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342392    5041 flags.go:64] FLAG: --lock-file=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342394    5041 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342397    5041 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342400    5041 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342406    5041 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342408    5041 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342411    5041 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342414    5041 flags.go:64] FLAG: --logging-format="text"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342417    5041 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342421    5041 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342423    5041 flags.go:64] FLAG: --manifest-url=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342427    5041 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342431    5041 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 19:58:32.350260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342435    5041 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342440    5041 flags.go:64] FLAG: --max-pods="110"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342443    5041 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342446    5041 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342449    5041 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342451    5041 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342454    5041 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342457    5041 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342460    5041 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342467    5041 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342470    5041 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342473    5041 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342476    5041 flags.go:64] FLAG: --pod-cidr=""
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342479    5041 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342482    5041 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342485    5041 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342488    5041 flags.go:64] FLAG: --port="10250"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342491    5041 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342494    5041 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06b4c5b941d71eb7d"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342497    5041 flags.go:64] FLAG: --qos-reserved=""
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342500    5041 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342503    5041 flags.go:64] FLAG: --register-node="true"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342506    5041 flags.go:64] FLAG: --register-with-taints=""
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342510    5041 flags.go:64] FLAG: --registry-burst="10"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342514    5041 flags.go:64] FLAG: --registry-qps="5"
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342517    5041 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 19:58:32.350904 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342520    5041 flags.go:64] FLAG: --reserved-memory=""
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342524    5041 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342527    5041 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342530    5041 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342533    5041 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342536    5041 flags.go:64] FLAG: --runonce="false"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342539    5041 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342544    5041 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342547    5041 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342550    5041 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342553    5041 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342556    5041 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342559    5041 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342562    5041 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342565    5041 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342568    5041 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342571    5041 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342574    5041 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342577    5041 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342579    5041 flags.go:64] FLAG: --system-cgroups=""
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342582    5041 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342587    5041 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342590    5041 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342593    5041 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342597    5041 flags.go:64] FLAG: --tls-min-version=""
Nov 01 19:58:32.351659 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342600    5041 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342603    5041 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342606    5041 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342608    5041 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342611    5041 flags.go:64] FLAG: --v="2"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342617    5041 flags.go:64] FLAG: --version="false"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342621    5041 flags.go:64] FLAG: --vmodule=""
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342625    5041 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.342628    5041 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342797    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342802    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342805    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342808    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342811    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342815    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342819    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342821    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342824    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342826    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342829    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342832    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.352289 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342834    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342837    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342840    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342842    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342844    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342847    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342850    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342852    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342855    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342857    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342860    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342863    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342867    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342871    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342874    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342877    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342882    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342885    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342888    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.352917 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342892    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342895    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342899    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342902    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342905    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342907    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342912    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342914    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342918    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342920    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342923    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342926    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342928    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342931    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342934    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342937    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342939    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342942    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342944    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.353555 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342947    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342949    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342952    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342954    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342957    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342959    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342962    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342964    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342967    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342969    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342974    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342976    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342979    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342982    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342984    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342987    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342989    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342992    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342995    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.342999    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.354274 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343002    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343004    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343008    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343011    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343013    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343016    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343018    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343021    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343023    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343025    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343028    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343031    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343033    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343036    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343038    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343041    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343043    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343046    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343048    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.354888 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343051    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343053    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343056    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343060    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343063    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343065    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343068    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343070    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343073    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343075    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343078    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343080    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343083    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343086    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343089    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343091    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.355416 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343094    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.343121    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343159    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343163    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343167    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343171    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343173    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343176    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343179    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343181    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343185    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343187    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343190    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343193    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343195    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343198    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.355954 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343200    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343203    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343206    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343208    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343212    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343215    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343217    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343220    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343223    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343225    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343228    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343230    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343233    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343235    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343238    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343241    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343243    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343246    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343249    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343251    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.356452 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343254    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343257    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343259    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343262    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343266    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343270    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343274    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343278    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343282    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343286    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343290    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343293    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343295    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343298    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343300    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343303    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343309    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343312    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343315    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343317    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.356996 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343320    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343322    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343325    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343328    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343331    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343333    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343336    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343338    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343341    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343344    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343347    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343350    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343352    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343355    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343358    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343360    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343363    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343365    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343368    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343370    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.357511 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343373    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343375    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343378    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343380    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343383    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343385    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343388    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343390    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343394    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343397    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343400    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343402    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343405    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343407    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343410    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343413    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343416    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343418    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343421    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.358065 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343423    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343427    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343430    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343432    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343435    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343438    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343440    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343443    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343447    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343451    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343454    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343456    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.343459    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.348395    5041 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.348412    5041 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.348429    5041 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.348434    5041 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348491    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348496    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.358516 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348499    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348502    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348504    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348507    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348510    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348512    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348515    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348518    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348521    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348524    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348526    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348529    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348532    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348534    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348537    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348540    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348543    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348546    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348549    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348551    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.359117 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348554    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348556    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348559    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348562    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348564    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348567    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348576    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348579    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348582    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348585    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348588    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348590    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348593    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348595    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348598    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348601    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348603    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348606    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348608    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.359922 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348611    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348613    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348616    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348618    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348621    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348623    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348626    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348628    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348631    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348633    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348636    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348640    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348644    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348647    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348650    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348653    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348656    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348660    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348663    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.360445 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348667    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348670    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348672    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348675    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348678    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348680    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348683    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348686    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348688    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348691    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348693    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348697    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348699    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348702    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348704    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348707    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348709    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348712    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348715    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348721    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.361062 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348724    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348727    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348729    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348732    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348734    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348737    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348740    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348742    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348745    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348747    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348750    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348754    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348757    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348759    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348762    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348765    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348767    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348770    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348772    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.361536 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348794    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348798    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348801    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348804    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348807    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348810    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348812    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.348840    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348972    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348977    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348980    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348983    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348985    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348989    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348992    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348994    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.362004 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.348997    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349000    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349002    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349005    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349007    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349010    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349013    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349016    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349019    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349021    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349024    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349027    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349029    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349032    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349035    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349037    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349040    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349042    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349045    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.362387 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349047    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349051    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349053    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349056    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349058    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349061    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349063    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349066    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349069    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349071    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349074    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349077    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349080    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349082    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349085    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349088    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349090    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349093    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349095    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.362861 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349098    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349101    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349104    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349107    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349110    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349113    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349115    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349118    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349120    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349123    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349125    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349128    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349131    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349133    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349136    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349138    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349141    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349143    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349146    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.363386 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349150    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349154    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349157    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349160    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349163    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349167    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349170    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349173    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349176    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349179    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349181    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349183    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349186    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349189    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349191    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349194    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349196    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349199    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349202    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349204    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.363870 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349207    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349209    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349212    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349214    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349217    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349219    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349222    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349225    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349227    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349230    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349232    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349234    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349237    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349239    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349242    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349244    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349247    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349249    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349252    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.364355 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349255    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349257    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.349280    5041 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349312    5041 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349316    5041 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349319    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349322    5041 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349325    5041 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349327    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349330    5041 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349333    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349335    5041 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349338    5041 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349340    5041 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349343    5041 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349345    5041 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 19:58:32.364889 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349348    5041 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349350    5041 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349353    5041 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349355    5041 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349358    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349360    5041 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349363    5041 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349366    5041 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349368    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349371    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349374    5041 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349376    5041 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349379    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349381    5041 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349383    5041 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349386    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349389    5041 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349392    5041 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349395    5041 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349397    5041 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 19:58:32.365342 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349400    5041 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349402    5041 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349405    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349407    5041 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349410    5041 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349412    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349415    5041 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349417    5041 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349420    5041 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349422    5041 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349425    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349427    5041 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349430    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349432    5041 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349435    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349437    5041 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349440    5041 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349447    5041 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349450    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 19:58:32.365845 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349453    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349455    5041 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349458    5041 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349461    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349463    5041 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349466    5041 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349469    5041 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349471    5041 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349474    5041 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349477    5041 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349481    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349487    5041 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349490    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349492    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349495    5041 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349497    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349501    5041 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349503    5041 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349506    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 19:58:32.366341 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349508    5041 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349511    5041 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349513    5041 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349516    5041 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349519    5041 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349521    5041 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349524    5041 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349527    5041 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349529    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349532    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349535    5041 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349539    5041 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349542    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349545    5041 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349549    5041 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349552    5041 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349554    5041 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349558    5041 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349562    5041 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 19:58:32.366825 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349565    5041 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349567    5041 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349570    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349572    5041 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349576    5041 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349579    5041 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349582    5041 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349585    5041 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349587    5041 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349590    5041 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349592    5041 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349596    5041 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349599    5041 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349601    5041 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349604    5041 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:32.349606    5041 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.349655    5041 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.350602    5041 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.350654    5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.351102    5041 server.go:1014] "Starting client certificate rotation"
Nov 01 19:58:32.367282 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.351251    5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.351307    5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:52:51 +0000 UTC" deadline="2028-07-11 03:14:47.22568009 +0000 UTC"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.351334    5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14815h16m14.874348461s"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.351547    5041 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.352689    5041 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.353952    5041 log.go:25] "Validated CRI v1 runtime API"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.355646    5041 log.go:25] "Validated CRI v1 image API"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.356361    5041 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 19:58:32.367837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.358360    5041 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b0333523-96f3-4b6c-bbfc-b656ffaeacc3:/dev/nvme0n1p4 dbf1dad0-27e6-4e89-9bd4-dae8b3ca0711:/dev/nvme0n1p3]
Nov 01 19:58:32.368077 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.358380    5041 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e156f9f8aa72ad595edd188f5ee5d207b5540ad3bad30b64919cf4a0420555f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e156f9f8aa72ad595edd188f5ee5d207b5540ad3bad30b64919cf4a0420555f/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a23a6ef453d8ce8f60a2f80877a9a759326861058322f770a0c4e59f4483e0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a23a6ef453d8ce8f60a2f80877a9a759326861058322f770a0c4e59f4483e0d/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/835afd727aad8d22650bfdc2c425d18a0ada2db54591acce6b4ff99d59955a48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/835afd727aad8d22650bfdc2c425d18a0ada2db54591acce6b4ff99d59955a48/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e1173f67c1aad4a20e4c989e6cb7ebff89ce0dbc710219213037e462ac6d300/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e1173f67c1aad4a20e4c989e6cb7ebff89ce0dbc710219213037e462ac6d300/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f41f87af22f6fdc0c5c32873ac96cdc9fe52c00391ca816e183ace2a4977f8f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f41f87af22f6fdc0c5c32873ac96cdc9fe52c00391ca816e183ace2a4977f8f2/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~projected/kube-api-access-69dg6:{mountpoint:/var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~projected/kube-api-access-69dg6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/087c078b-50e9-4942-ac02-6faf11ffecf3/volumes/kubernetes.io~projected/kube-api-access-kjb2f:{mountpoint:/var/lib/kubelet/pods/087c078b-50e9-4942-ac02-6faf11ffecf3/volumes/kubernetes.io~projected/kube-api-access-kjb2f major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a1313c-bd53-44ed-8984-6fc0f890fc3c/volumes/kubernetes.io~projected/kube-api-access-s2s7r:{mountpoint:/var/lib/kubelet/pods/25a1313c-bd53-44ed-8984-6fc0f890fc3c/volumes/kubernetes.io~projected/kube-api-access-s2s7r major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b1aad29-2c51-448a-92cd-abc01fe696d1/volumes/kubernetes.io~projected/kube-api-access-znzbc:{mountpoint:/var/lib/kubelet/pods/2b1aad29-2c51-448a-92cd-abc01fe696d1/volumes/kubernetes.io~projected/kube-api-access-znzbc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fad2018-8963-46b5-a399-e165e48dde59/volumes/kubernetes.io~projected/kube-api-access-xdbfs:{mountpoint:/var/lib/kubelet/pods/2fad2018-8963-46b5-a399-e165e48dde59/volumes/kubernetes.io~projected/kube-api-access-xdbfs major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/379fc616-e382-4250-8ec7-34217917ea30/volumes/kubernetes.io~projected/kube-api-access-n8gr9:{mountpoint:/var/lib/kubelet/pods/379fc616-e382-4250-8ec7-34217917ea30/volumes/kubernetes.io~projected/kube-api-access-n8gr9 major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~projected/kube-api-access-5bvt6:{mountpoint:/var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~projected/kube-api-access-5bvt6 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~projected/kube-api-access-rwj62:{mountpoint:/var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~projected/kube-api-access-rwj62 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~secret/signing-key major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bbad68d-9021-4665-9315-d4ac8521f567/volumes/kubernetes.io~projected/kube-api-access-chv5d:{mountpoint:/var/lib/kubelet/pods/5bbad68d-9021-4665-9315-d4ac8521f567/volumes/kubernetes.io~projected/kube-api-access-chv5d major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/kube-api-access-dg49c:{mountpoint:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/kube-api-access-dg49c major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6533ed5f-4009-45be-8d6f-b90f1848d640/volumes/kubernetes.io~projected/kube-api-access-z45gb:{mountpoint:/var/lib/kubelet/pods/6533ed5f-4009-45be-8d6f-b90f1848d640/volumes/kubernetes.io~projected/kube-api-access-z45gb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fb413b1-e547-411a-a27a-50716d2bd1ef/volumes/kubernetes.io~projected/kube-api-access-j9bgd:{mountpoint:/var/lib/kubelet/pods/6fb413b1-e547-411a-a27a-50716d2bd1ef/volumes/kubernetes.io~projected/kube-api-access-j9bgd major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4/volumes/kubernetes.io~projected/kube-api-access-bbkc5:{mountpoint:/var/lib/kubelet/pods/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4/volumes/kubernetes.io~projected/kube-api-access-bbkc5 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~projected/kube-api-access-wjwn8:{mountpoint:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~projected/kube-api-access-wjwn8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f68e3cb-db93-4919-acb4-b60652805f67/volumes/kubernetes.io~projected/kube-api-access-lk5lx:{mountpoint:/var/lib/kubelet/pods/7f68e3cb-db93-4919-acb4-b60652805f67/volumes/kubernetes.io~projected/kube-api-access-lk5lx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4/volumes/kubernetes.io~projected/kube-api-access-j75gk:{mountpoint:/var/lib/kubelet/pods/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4/volumes/kubernetes.io~projected/kube-api-access-j75gk major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8961d680-2f89-4358-a9fc-ff43d01e2eda/volumes/kubernetes.io~projected/kube-api-access-tvgtp:{mountpoint:/var/lib/kubelet/pods/8961d680-2f89-4358-a9fc-ff43d01e2eda/volumes/kubernetes.io~projected/kube-api-access-tvgtp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91b01765-cd4d-46bb-b53e-54d52f846a57/volumes/kubernetes.io~projected/kube-api-access-l5f4b:{mountpoint:/var/lib/kubelet/pods/91b01765-cd4d-46bb-b53e-54d52f846a57/volumes/kubernetes.io~projected/kube-api-access-l5f4b major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9666c411-ce1d-4592-82de-51b88b02ea47/volumes/kubernetes.io~projected/kube-api-access-gnfpz:{mountpoint:/var/lib/kubelet/pods/9666c411-ce1d-4592-82de-51b88b02ea47/volumes/kubernetes.io~projected/kube-api-access-gnfpz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~projected/kube-api-access-bx67m:{mountpoint:/var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~projected/kube-api-access-bx67m major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1c41cc7-e221-4bc2-aec2-af48980dc5eb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a1c41cc7-e221-4bc2-aec2-af48980dc5eb/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~projected/kube-api-access-xxg5c:{mountpoint:/var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~projected/kube-api-access-xxg5c major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7cde0f8-1666-42a5-84e1-34f1ab468d1e/volumes/kubernetes.io~projected/kube-api-access-q5wlz:{mountpoint:/var/lib/kubelet/pods/a7cde0f8-1666-42a5-84e1-34f1ab468d1e/volumes/kubernetes.io~projected/kube-api-access-q5wlz major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab5534dc-85a6-4a5f-8101-7fc41ef4999e/volumes/kubernetes.io~projected/kube-api-access-mdz7p:{mountpoint:/var/lib/kubelet/pods/ab5534dc-85a6-4a5f-8101-7fc41ef4999e/volumes/kubernetes.io~projected/kube-api-access-mdz7p major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acc92340-bb89-4ff1-838e-08a6574531b9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/acc92340-bb89-4ff1-838e-08a6574531b9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~projected/kube-api-access-8ljsg:{mountpoint:/var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~projected/kube-api-access-8ljsg major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~projected/kube-api-access-bx5hj:{mountpoint:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~projected/kube-api-access-bx5hj major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~projected/kube-api-access-66grf:{mountpoint:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~projected/kube-api-access-66grf major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c769b88a-d848-424a-bacb-79d80ea49e48/volumes/kubernetes.io~projected/kube-api-access-4p4hf:{mountpoint:/var/lib/kubelet/pods/c769b88a-d848-424a-bacb-79d80ea49e48/volumes/kubernetes.io~projected/kube-api-access-4p4hf major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/kube-api-access-ffqsb:{mountpoint:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/kube-api-access-ffqsb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbb64076-bc8a-41d9-86b7-179dba150c84/volumes/kubernetes.io~projected/kube-api-access-v2bkf:{mountpoint:/var/lib/kubelet/pods/cbb64076-bc8a-41d9-86b7-179dba150c84/volumes/kubernetes.io~projected/kube-api-access-v2bkf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efd428db-ed11-4811-b94d-76b9e12e4288/volumes/kubernetes.io~projected/kube-api-access-bj5s2:{mountpoint:/var/lib/kubelet/pods/efd428db-ed11-4811-b94d-76b9e12e4288/volumes/kubernetes.io~projected/kube-api-access-bj5s2 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/volumes/kubernetes.io~projected/kube-api-access-pzdb9:{mountpoint:/var/lib/kubelet/pods/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/volumes/kubernetes.io~projected/kube-api-access-pzdb9 major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c2ba7ba861fc4632c03cb7d6547c7eda8be3366a6ded77b4e0d7a811b0cecc03/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e23db4747b424e29e991c78ad4afb630e9f47f389392ff49dcaa1b42324ff91b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7ef2f74544bd539b5242ac1289c32a41de1709485e59cfa66c668a81b483fb32/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a8ad97c94dc991d31befae79b408113ffb5e3ec4b28f9370b00be5efe0892090/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/63ef694847d51044cc7fcee232533b6296b178e162dff79df8ce2e1419f677c9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/112a9ba0f27980abb4cca0100ee32b30d4f61256eb997026f557a597fe84bfa4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9ac64e08977b6d0cc2b56385093b741d6ef83aea674ad2507ad10d1f7fc33cd8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6aa8634cb50f964e4cf9f12ad8d227f0d4c58af47b17a7d81f1a236673296394/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/74855f7170993bfb4dcf654899f8d760c27572b849b092b73cc3d84f17461d4e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/625f177042225aa38c51de8401d748a644d5856f89193279c9af7abb9d345ff3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/129a49903d2597a821f3d0d0a0b663044f33efe29b756e223f43e9b9b9a6a02b/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/be00a56bfc2c29d57993d01f27d592f3b5243135fe0d8d40bace0130aa7bdb11/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e1f006e2d6307cf679ff65854ffbfe6d5e4e519d7126e0c8b806331cba1e6658/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f5a48b235dbe6e5101eed0cddbadcb6d6030283c2f1fa25c103e14399bd74511/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/21c7b183e51d89b1128b0a33843dd9f6bdae4c9d83b26d97e9b0851b0f6a089b/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/14539568c2ccc825d8b5b37ab304462ab705905fd2aa1e2170f2419b83ceb1e1/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/131d5fc6552cb4c709498007430cab4045a8e1226edf268cf37386e3e709165c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4992ba7d4034c476a430b4e0a4b2298de87074f45798f709c21ce7eb16e700f0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d0a068707b934c8df50a697f1236429dea322c75dc8dbb841ae3c4244c2be098/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/45f4d4b9d2ddb7cdc36209edefad07314ae08585ae1ff03bc1914ae73e68c578/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e763288990b5f78885c20eecb4b58858370a2e7adce82750e3ea962bd1655930/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c3f36eb3c64750f6dae65b82cfafdb840ce5e6522ff8cc23667cf7ba811b8a7d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/bf39a64348716933947172f6486f58bc8e7627e7d932251cb7f200959a1ebaa0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e9d3a76e24ef99295d85e97bf5e91e660fda93b9505937eeb0c43194606d52f1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/12de0a92a6380cf4711124424b1aa8c863f371ed6592289ab95d7530a5a75b9f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/87384e06495f5e10c4f8291e272e3b1a6d1c227b88fae31d661efc30ca538d9a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8aa91abd28383a74f0a76e58c80d21617eba9b07dbb944e3582f97f2b0dc87ba/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f0166ce83a88fbf6acaa8a01e565a9704a9c51941604e9f3ff7ededcb6f91e2e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/76517eb9517d5859ddd4fea1926c68574715191d80063c7a478f73c4f02f2271/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/f9f9cb12106ddce18107e686536ebeb136b567e0620452e4008e68034b3647d0/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e54c3f6804242ddc69c385136bd4ec3cff34e662402f4c1399e9416298b371a0/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/66db38d3ea44ffa90764d8f887b5a53bcb0cda6038b2a0e4751b64bc253719e3/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/e4f680efb6dce7db6959fd3f43c40cf43355992a9903e24dd25804b1e32d3fda/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/75a7f576aa1e044c15a821f3eb966a9e11b13f8474d6e40ff0caeb82870d2f0f/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/7ed317792e08eb1c40277a25d6c140976de056a0d5cd620f898fe9433f24a307/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/00c25e5c36f6d2bf5a7523dd22f8b6a0098ce867c0904e2e75d3003f441e9b27/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8823cc3c2ced8441033e17aec7777400f84ea934a8dca43f0dc797519df74799/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/17f47c3b6f4d8f4746e42fa7ea6ce1497041da78dacfe85f77a7127f37b5dc23/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/fd757a108aeb533e4a5dae031206c117712c938b96a36e53ca54d6d80e2c3541/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/6c6b1f0d86dc09530f3d32277683c17bb3e2c2958fe1f915b8a52b290fd86fc9/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/7afb45b3e6db02d1b47ff35841ed887e1fc5b6b6211601cc0ed72140bf2ea250/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/cd4bef1bf5f983f100db18ab09fd3f6aad77fceb34397526a1451cda57de2923/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/db9c0acb951ab13cbae1520a6c998522498fad2baf6c25898b52c20a4f1a43cd/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/554d8b347133798d1ee97a5cc30cf54dd83ba54f791dead4e8e8d2b8a59d89cd/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/816df9a7a92eca452eacf83df88174ade81c40ceb26002f6642c7cb3f1c19f4b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/58dcbf945a876f0d146b0ce1db60ae89338a2ca75b8682f21127f3e1d522b81a/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/8303be5f97fe5b873382927adf9d21a828ef8b6a92917e6367577554d660b21f/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/5ea984c573c6b6c6d971958bcfd2950775f62c169ea99f18c641281c38f007bd/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/a42a453b89958764cb5bbd164111be701b5ac975824fdcff6f484dce4a542564/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/370a20f693d23bb73a05c3094b07e416ec0cda0916e55745a1509fcfc584eb4f/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/be8a3e5f32ba30c49cd459e3c0ca221f1d342615bb5990f64a6922ef7fb35984/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/b8d8e0cb490381cb51f051c27187fed6188aa253d14511dac14996285b49edb8/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f9eeb92861ccff28ba879a5390dd15237717dabf7d0885db10044d9c33fec52c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d9a56651826c936291543824ac60425c8f06767e995b672b167da6993262c60f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/bd374bdeb96fc8de0e694f2361ad5154a99a2f4fcbb8799efd6e6cb339771867/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/9c967c68d666584935aa88dd2087b39133bd40765b781964fd64740b5ae0fcd3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6611e6fe87a44f74add8d8278ab36c812aeeb3a132969c1aa69ab38b10a25c60/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0f23878997ef57eb604bbb81acbc76bc25e346720dea00bdbfd574644587a8ab/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b52982906b9d6578eeb37410a766fdc673230a351a839b54c6fd69484428308f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/99ffc1236538a1b04fb3702503c95b9e417b0d8b0497a0789f75cdb1b638b49e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/292a15f128c3637fba20e27686b35529ecd1d24c0449beb39bf02ba7948e3a5b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/226c06b6bf1ad7eff137250cfa41d14f5800db794ce63620d64f32fa18086a46/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 01 19:58:32.368296 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.367592    5041 manager.go:217] Machine: {Timestamp:2026-11-01 19:58:32.36712829 +0000 UTC m=+0.088206957 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100770 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28bd46c15cb0b55257f8f00f37dc2f SystemUUID:ec28bd46-c15c-b0b5-5257-f8f00f37dc2f BootID:15d6de52-f155-4dd1-9656-f638a82e6068 Filesystems:[{Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/volumes/kubernetes.io~projected/kube-api-access-pzdb9 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5ce6081c47354471a3c9887335c382ea72e6c23e2acb6a4b23189b75a5b267d/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f594bc97a01c2ad2f6e8e118c1ec3ae07a9085a024e93a95588b981cb3bc855/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a2986c6152f34c6de790ba68f4c88f54569993c8b3880967aaaee2190138246/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a2dbcb461e5e33b2f2d69cbfdec65cdd413b81eb01324c813677ead0e5d0007/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a23a6ef453d8ce8f60a2f80877a9a759326861058322f770a0c4e59f4483e0d/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbb64076-bc8a-41d9-86b7-179dba150c84/volumes/kubernetes.io~projected/kube-api-access-v2bkf DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~projected/kube-api-access-wjwn8 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/kube-api-access-dg49c DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~projected/kube-api-access-66grf DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5a82104f86ab093d416574900ae73bb01876b605e97c43278cc6f618cb4295c/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1c41cc7-e221-4bc2-aec2-af48980dc5eb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e1173f67c1aad4a20e4c989e6cb7ebff89ce0dbc710219213037e462ac6d300/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efd428db-ed11-4811-b94d-76b9e12e4288/volumes/kubernetes.io~projected/kube-api-access-bj5s2 DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d887126-1d78-4f44-afaa-28b9d682e211/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcdce30b9b42fcc713c215430c501c4dd3ab8ff5c33e8375360130742871dc60/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/25a1313c-bd53-44ed-8984-6fc0f890fc3c/volumes/kubernetes.io~projected/kube-api-access-s2s7r DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/087c078b-50e9-4942-ac02-6faf11ffecf3/volumes/kubernetes.io~projected/kube-api-access-kjb2f DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3959347d1fa75fe7b2e15ca4883ef0a20463023949af9d34739f4805f7ff640/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab5534dc-85a6-4a5f-8101-7fc41ef4999e/volumes/kubernetes.io~projected/kube-api-access-mdz7p DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~projected/kube-api-access-ffqsb DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ab485a2f73772ef99f7a8f7e83f92ca5baa10fcb3004027ab7c1afb32258a07/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f41f87af22f6fdc0c5c32873ac96cdc9fe52c00391ca816e183ace2a4977f8f2/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e823a375128c77915c90ec98012225a9e1fbf616231d74e52a18f3f2ca1f71e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b456599-bf71-4bab-afa7-e2978c6e4e42/volumes/kubernetes.io~projected/kube-api-access-bx67m DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bbad68d-9021-4665-9315-d4ac8521f567/volumes/kubernetes.io~projected/kube-api-access-chv5d DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50b42c9bb9a5229b242ac72e0f131e7f50cf5a92c10d4ca169f40cd9dbfc9887/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3740c37dae8234daf8a5f4a8e03fdeb61209fef7e1dc6ac8a617de8b09837e00/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:354 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/adfbd1ed-3a02-4334-b370-328f2a511729/volumes/kubernetes.io~projected/kube-api-access-8ljsg DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7cde0f8-1666-42a5-84e1-34f1ab468d1e/volumes/kubernetes.io~projected/kube-api-access-q5wlz DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~projected/kube-api-access-69dg6 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46c95e40-824d-468a-99f4-57ec8b88fb42/volumes/kubernetes.io~projected/kube-api-access-5bvt6 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4/volumes/kubernetes.io~projected/kube-api-access-bbkc5 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e156f9f8aa72ad595edd188f5ee5d207b5540ad3bad30b64919cf4a0420555f/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91b01765-cd4d-46bb-b53e-54d52f846a57/volumes/kubernetes.io~projected/kube-api-access-l5f4b DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/544b6ff8-e957-4b48-880f-2311ad4aba68/volumes/kubernetes.io~projected/kube-api-access-rwj62 DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8961d680-2f89-4358-a9fc-ff43d01e2eda/volumes/kubernetes.io~projected/kube-api-access-tvgtp DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c14186d7-dc3e-4df1-9358-806da5cbc684/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1aa81611eb313be16effcfaa5995c131b004809c72457a64587e85cd4ce1d614/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/715f7ea1adc396bdb04f26a759e41dab6217705de6756540b4644ad6027c51f9/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53ffc120f4023c0b180d1b57d2ab56a5d07786435d4313365d932472568fb20b/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6fb413b1-e547-411a-a27a-50716d2bd1ef/volumes/kubernetes.io~projected/kube-api-access-j9bgd DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6533ed5f-4009-45be-8d6f-b90f1848d640/volumes/kubernetes.io~projected/kube-api-access-z45gb DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b1aad29-2c51-448a-92cd-abc01fe696d1/volumes/kubernetes.io~projected/kube-api-access-znzbc DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4/volumes/kubernetes.io~projected/kube-api-access-j75gk DeviceMajor:0 DeviceMinor:334 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f68e3cb-db93-4919-acb4-b60652805f67/volumes/kubernetes.io~projected/kube-api-access-lk5lx DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb48980dc5fdde7923267ce5df4ea164e2bc937acac03176390738711dce84b3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/835afd727aad8d22650bfdc2c425d18a0ada2db54591acce6b4ff99d59955a48/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34714b3824e89a84f2885082862244b293205579df1fa198252a34a31eeef587/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9666c411-ce1d-4592-82de-51b88b02ea47/volumes/kubernetes.io~projected/kube-api-access-gnfpz DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2fad2018-8963-46b5-a399-e165e48dde59/volumes/kubernetes.io~projected/kube-api-access-xdbfs DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4767580ca45971a9ed32a6e522ecece9d2e077110d6b9383b6ac34ef11c2456f/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acc92340-bb89-4ff1-838e-08a6574531b9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a5aab77d-3a73-46b8-ba03-03e96ebf16f0/volumes/kubernetes.io~projected/kube-api-access-xxg5c DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c769b88a-d848-424a-bacb-79d80ea49e48/volumes/kubernetes.io~projected/kube-api-access-4p4hf DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c020f3c7-45a8-4226-9e41-289a4a7ab737/volumes/kubernetes.io~projected/kube-api-access-bx5hj DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dea548100bb55ae09aeca9972b12c9b30b610b3c4a34272533931e902df4243/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/379fc616-e382-4250-8ec7-34217917ea30/volumes/kubernetes.io~projected/kube-api-access-n8gr9 DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1aa81611eb313be MacAddress:6a:ad:0a:59:13:7b Speed:10000 Mtu:8901} {Name:2657699e1be90d0 MacAddress:62:62:8e:cb:f1:a4 Speed:10000 Mtu:8901} {Name:2ab485a2f73772e MacAddress:a6:24:4e:92:32:1c Speed:10000 Mtu:8901} {Name:34714b3824e89a8 MacAddress:b2:88:10:35:c8:aa Speed:10000 Mtu:8901} {Name:3740c37dae8234d MacAddress:fe:ff:5f:4f:85:47 Speed:10000 Mtu:8901} {Name:4767580ca45971a MacAddress:7e:52:eb:a8:a4:43 Speed:10000 Mtu:8901} {Name:4dea548100bb55a MacAddress:9e:f9:3c:06:d7:6e Speed:10000 Mtu:8901} {Name:50b42c9bb9a5229 MacAddress:1a:f0:77:28:47:dc Speed:10000 Mtu:8901} {Name:5e156f9f8aa72ad MacAddress:a6:f5:c6:c4:72:17 Speed:10000 Mtu:8901} {Name:5e823a375128c77 MacAddress:0e:71:ec:db:dc:75 Speed:10000 Mtu:8901} {Name:715f7ea1adc396b MacAddress:fe:4b:d2:82:0c:88 Speed:10000 Mtu:8901} {Name:7a23a6ef453d8ce MacAddress:82:ef:33:0b:71:96 Speed:10000 Mtu:8901} {Name:b3959347d1fa75f MacAddress:1a:ac:c8:28:70:95 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e7:ef:95:21 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:c8:43:cf:c1:6f Speed:0 Mtu:8901} {Name:e5a82104f86ab09 MacAddress:fa:f7:b0:49:e3:37 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e7:ef:95:21 Speed:0 Mtu:9001} {Name:f41f87af22f6fdc MacAddress:1a:c2:9a:96:ef:7b Speed:10000 Mtu:8901} {Name:fcdce30b9b42fcc MacAddress:0e:cd:62:cc:46:98 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:86:fd:3d:03:80:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:99:4b:40:46:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 19:58:32.368296 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368191    5041 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 19:58:32.368296 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368251    5041 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 19:58:32.368650 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368622    5041 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 19:58:32.368792 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368650    5041 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-175.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 19:58:32.368859 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368800    5041 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 19:58:32.368859 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368809    5041 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 19:58:32.368859 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368820    5041 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:58:32.368859 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368832    5041 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 19:58:32.368963 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.368924    5041 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 19:58:32.369112 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369102    5041 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 19:58:32.369215 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369206    5041 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 19:58:32.369245 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369240    5041 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 19:58:32.369272 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369253    5041 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 19:58:32.369272 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369271    5041 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 19:58:32.369323 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369280    5041 kubelet.go:405] "Adding apiserver pod source"
Nov 01 19:58:32.369323 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.369288    5041 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 19:58:32.370444 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.370429    5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 19:58:32.370515 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.370456    5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:58:32.371056 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.371037    5041 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 19:58:32.374347 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374330    5041 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 19:58:32.374347 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374343    5041 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 19:58:32.374497 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374487    5041 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 19:58:32.374691 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374680    5041 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 19:58:32.374736 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374717    5041 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 19:58:32.374829 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374819    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374834    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374840    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374847    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374853    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374859    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374865    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 19:58:32.374870 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374870    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374878    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374884    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374892    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374901    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374922    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 19:58:32.375109 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.374928    5041 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 19:58:32.378421 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.378408    5041 server.go:1263] "Started kubelet"
Nov 01 19:58:32.379405 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.379273    5041 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 19:58:32.379461 ip-10-0-137-175 systemd[1]: Started Kubernetes Kubelet.
Nov 01 19:58:32.379600 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.378559    5041 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 19:58:32.379655 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.379604    5041 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 19:58:32.384817 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.379924    5041 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 19:58:32.384817 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.380746    5041 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:32.384817 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.381078    5041 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 19:58:32.384817 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.383736    5041 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:32.386556 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386535    5041 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 19:58:32.386676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386560    5041 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 19:58:32.386676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386642    5041 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:52:53 +0000 UTC" deadline="2028-05-08 18:52:43.140061045 +0000 UTC"
Nov 01 19:58:32.386676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386658    5041 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 19:58:32.386676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386663    5041 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13294h54m10.753400665s"
Nov 01 19:58:32.386676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386674    5041 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 19:58:32.386988 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.386814    5041 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 19:58:32.389446 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389399    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b456599-bf71-4bab-afa7-e2978c6e4e42" volumeName="kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m" seLinuxMountContext=""
Nov 01 19:58:32.389538 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389452    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c020f3c7-45a8-4226-9e41-289a4a7ab737" volumeName="kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj" seLinuxMountContext=""
Nov 01 19:58:32.389538 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389467    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a1313c-bd53-44ed-8984-6fc0f890fc3c" volumeName="kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r" seLinuxMountContext=""
Nov 01 19:58:32.389538 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389480    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp" seLinuxMountContext=""
Nov 01 19:58:32.389538 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389520    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg" seLinuxMountContext=""
Nov 01 19:58:32.389538 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389533    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbb64076-bc8a-41d9-86b7-179dba150c84" volumeName="kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389546    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" volumeName="kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389558    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" volumeName="kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389575    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389587    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389601    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7cde0f8-1666-42a5-84e1-34f1ab468d1e" volumeName="kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389619    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acc92340-bb89-4ff1-838e-08a6574531b9" volumeName="kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389632    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389645    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efd428db-ed11-4811-b94d-76b9e12e4288" volumeName="kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389667    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46c95e40-824d-468a-99f4-57ec8b88fb42" volumeName="kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389680    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bbad68d-9021-4665-9315-d4ac8521f567" volumeName="kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d" seLinuxMountContext=""
Nov 01 19:58:32.389699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389694    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d887126-1d78-4f44-afaa-28b9d682e211" volumeName="kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389708    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f68e3cb-db93-4919-acb4-b60652805f67" volumeName="kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389721    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91b01765-cd4d-46bb-b53e-54d52f846a57" volumeName="kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389733    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbb64076-bc8a-41d9-86b7-179dba150c84" volumeName="kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389745    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867" volumeName="kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389757    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544b6ff8-e957-4b48-880f-2311ad4aba68" volumeName="kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389771    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b456599-bf71-4bab-afa7-e2978c6e4e42" volumeName="kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389803    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14186d7-dc3e-4df1-9358-806da5cbc684" volumeName="kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389815    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a1313c-bd53-44ed-8984-6fc0f890fc3c" volumeName="kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389827    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389839    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6533ed5f-4009-45be-8d6f-b90f1848d640" volumeName="kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389850    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9666c411-ce1d-4592-82de-51b88b02ea47" volumeName="kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389870    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c41cc7-e221-4bc2-aec2-af48980dc5eb" volumeName="kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389881    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389893    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c020f3c7-45a8-4226-9e41-289a4a7ab737" volumeName="kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389904    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c769b88a-d848-424a-bacb-79d80ea49e48" volumeName="kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389917    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46c95e40-824d-468a-99f4-57ec8b88fb42" volumeName="kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389929    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46c95e40-824d-468a-99f4-57ec8b88fb42" volumeName="kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389942    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4" volumeName="kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389955    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9666c411-ce1d-4592-82de-51b88b02ea47" volumeName="kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389967    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b456599-bf71-4bab-afa7-e2978c6e4e42" volumeName="kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389980    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.389993    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544b6ff8-e957-4b48-880f-2311ad4aba68" volumeName="kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390007    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d887126-1d78-4f44-afaa-28b9d682e211" volumeName="kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390021    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390033    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14186d7-dc3e-4df1-9358-806da5cbc684" volumeName="kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390045    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c769b88a-d848-424a-bacb-79d80ea49e48" volumeName="kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390058    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" volumeName="kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390073    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a1313c-bd53-44ed-8984-6fc0f890fc3c" volumeName="kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390085    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867" volumeName="kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390101    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="087c078b-50e9-4942-ac02-6faf11ffecf3" volumeName="kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390116    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6533ed5f-4009-45be-8d6f-b90f1848d640" volumeName="kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390128    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fb413b1-e547-411a-a27a-50716d2bd1ef" volumeName="kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390142    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f68e3cb-db93-4919-acb4-b60652805f67" volumeName="kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390155    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91b01765-cd4d-46bb-b53e-54d52f846a57" volumeName="kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b" seLinuxMountContext=""
Nov 01 19:58:32.390162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390170    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5aab77d-3a73-46b8-ba03-03e96ebf16f0" volumeName="kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390204    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390219    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c020f3c7-45a8-4226-9e41-289a4a7ab737" volumeName="kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390233    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b1aad29-2c51-448a-92cd-abc01fe696d1" volumeName="kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390241    5041 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390246    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f68e3cb-db93-4919-acb4-b60652805f67" volumeName="kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390290    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8961d680-2f89-4358-a9fc-ff43d01e2eda" volumeName="kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390308    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2ebd17c-0816-4e50-b17b-bb1ca3633730" volumeName="kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390322    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390335    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390347    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390360    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d887126-1d78-4f44-afaa-28b9d682e211" volumeName="kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390375    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5aab77d-3a73-46b8-ba03-03e96ebf16f0" volumeName="kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390388    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14186d7-dc3e-4df1-9358-806da5cbc684" volumeName="kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390401    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c769b88a-d848-424a-bacb-79d80ea49e48" volumeName="kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390413    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="379fc616-e382-4250-8ec7-34217917ea30" volumeName="kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390424    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d887126-1d78-4f44-afaa-28b9d682e211" volumeName="kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390435    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="888a6838-2cc9-4bc5-9ca0-52f1f4052ef4" volumeName="kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390448    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab5534dc-85a6-4a5f-8101-7fc41ef4999e" volumeName="kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390461    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390484    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c769b88a-d848-424a-bacb-79d80ea49e48" volumeName="kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390496    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390511    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="379fc616-e382-4250-8ec7-34217917ea30" volumeName="kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390524    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fad2018-8963-46b5-a399-e165e48dde59" volumeName="kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390539    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390550    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390562    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c41cc7-e221-4bc2-aec2-af48980dc5eb" volumeName="kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390574    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfbd1ed-3a02-4334-b370-328f2a511729" volumeName="kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390616    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" volumeName="kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390633    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d887126-1d78-4f44-afaa-28b9d682e211" volumeName="kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390650    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7cde0f8-1666-42a5-84e1-34f1ab468d1e" volumeName="kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390664    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" volumeName="kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390677    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a1313c-bd53-44ed-8984-6fc0f890fc3c" volumeName="kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390690    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="544b6ff8-e957-4b48-880f-2311ad4aba68" volumeName="kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390704    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60703b7e-ae49-4bec-923c-73ade0101e57" volumeName="kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390727    5041 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6533ed5f-4009-45be-8d6f-b90f1848d640" volumeName="kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir" seLinuxMountContext=""
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390742    5041 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 19:58:32.391151 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.390750    5041 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 19:58:32.392398 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392366    5041 factory.go:153] Registering CRI-O factory
Nov 01 19:58:32.392398 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392385    5041 factory.go:223] Registration of the crio container factory successfully
Nov 01 19:58:32.392523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392444    5041 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 19:58:32.392523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392453    5041 factory.go:55] Registering systemd factory
Nov 01 19:58:32.392523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392461    5041 factory.go:223] Registration of the systemd container factory successfully
Nov 01 19:58:32.392523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392482    5041 factory.go:103] Registering Raw factory
Nov 01 19:58:32.392523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392495    5041 manager.go:1199] Started watching for new ooms in manager
Nov 01 19:58:32.392860 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.392848    5041 manager.go:319] Starting recovery of all containers
Nov 01 19:58:32.393879 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.393864    5041 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 19:58:32.406419 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.406396    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:32.406641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.406620    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:32.418533 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.418506    5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 19:58:32.419839 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.419820    5041 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 19:58:32.419839 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.419843    5041 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 19:58:32.419968 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.419861    5041 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 19:58:32.419968 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:32.419895    5041 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 19:58:32.423260 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.423236    5041 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 19:58:32.430911 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.430891    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:32.430988 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.430919    5041 generic.go:356] "Generic (PLEG): container finished" podID="053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb" exitCode=255
Nov 01 19:58:32.441990 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.441959    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="f8ff7a56188eb127004f99916300e77097f629f78f85f29e561ec8ceb683fc22" exitCode=0
Nov 01 19:58:32.441990 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.441989    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="6f0ecb62906b93669dde4a64dde4242e6b30a32145c1ac3c6fac10e363656199" exitCode=0
Nov 01 19:58:32.442115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.442004    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="c07a2593be736e9924acc4267d4a162e9ac2672237f5947b87c91900b41c6c42" exitCode=0
Nov 01 19:58:32.442115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.442013    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="900c2ddc585fe6568905524e64b281e9489c402c07ac4eb1fb22b734ce4ecdda" exitCode=0
Nov 01 19:58:32.442115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.442022    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="949bdb4c0cd3867cbf0aff4e165826ea4c60d8b165726a91b23990e582ac403a" exitCode=0
Nov 01 19:58:32.442115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.442030    5041 generic.go:356] "Generic (PLEG): container finished" podID="25a1313c-bd53-44ed-8984-6fc0f890fc3c" containerID="146fd66eec32c00574298c9e34fe43dcd2faab3dfce09c2dd5de875b7dc41ffc" exitCode=0
Nov 01 19:58:32.463122 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.463089    5041 generic.go:356] "Generic (PLEG): container finished" podID="c5bb0f42cc122a3cebecd048bef257a8" containerID="c448c93da915add7c3239b421da195658cebc36dd393beb1b2bba26a70c0f1c4" exitCode=0
Nov 01 19:58:32.487332 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.487275    5041 manager.go:324] Recovery completed
Nov 01 19:58:32.509301 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509275    5041 cpu_manager.go:226] "Starting" policy="none"
Nov 01 19:58:32.509301 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509298    5041 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 19:58:32.509483 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509320    5041 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 19:58:32.509526 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509491    5041 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 19:58:32.509610 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509507    5041 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 19:58:32.509699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509614    5041 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 19:58:32.509699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509643    5041 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 19:58:32.509874 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509860    5041 policy_none.go:50] "Start"
Nov 01 19:58:32.509911 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509881    5041 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 19:58:32.509911 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.509897    5041 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 19:58:32.510056 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.510044    5041 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 19:58:32.510056 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.510054    5041 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 19:58:32.510128 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.510061    5041 policy_none.go:44] "Start"
Nov 01 19:58:32.514515 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.514489    5041 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 19:58:32.518066 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.518055    5041 manager.go:341] "Starting Device Plugin manager"
Nov 01 19:58:32.518138 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:32.518077    5041 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 19:58:32.518138 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.518086    5041 server.go:87] "Starting device plugin registration server"
Nov 01 19:58:32.519060 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519043    5041 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 19:58:32.519132 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519063    5041 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 19:58:32.519182 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519140    5041 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 19:58:32.519289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519256    5041 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 19:58:32.519373 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519302    5041 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 19:58:32.519373 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519309    5041 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 19:58:32.519462 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.519359    5041 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:58:32.519285301Z","UUID":"bf3154cd-dbcf-4712-aa30-eba106a44405","Handler":null,"Name":"","Endpoint":""}
Nov 01 19:58:32.520165 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.520123    5041 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"]
Nov 01 19:58:32.520899 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.520878    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.520985 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.520946    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"02aa65d6c5643e7fc09a137b81d3ea6b","Type":"ContainerStarted","Data":"b968fb6a70d7a12d9bff9e54eec47a62c99ee7afb2b4fa4244dc6452f5c2527c"}
Nov 01 19:58:32.521059 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521003    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"02aa65d6c5643e7fc09a137b81d3ea6b","Type":"ContainerStarted","Data":"e5a99f0a2d0b3918ca5319883d4f33767624aa4a2332dd5b8095feda5ba8ada6"}
Nov 01 19:58:32.521059 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521036    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerStarted","Data":"6927e1ece8f53e3036193fcda0b3f5861586c53aaf28c20be614fa1e3c4d6d31"}
Nov 01 19:58:32.521154 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521057    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerDied","Data":"c448c93da915add7c3239b421da195658cebc36dd393beb1b2bba26a70c0f1c4"}
Nov 01 19:58:32.521154 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521078    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal" event={"ID":"c5bb0f42cc122a3cebecd048bef257a8","Type":"ContainerStarted","Data":"66c78a91a8cdcc304a335d097a2b17a6659ae8027fb0d6400ded90acc86191c6"}
Nov 01 19:58:32.521154 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:32.521092    5041 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 19:58:32.521310 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521173    5041 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 19:58:32.521310 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521224    5041 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 19:58:32.521310 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.521297    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.533265 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.533243    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:58:32.533346 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:32.533302    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.533872 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.533860    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:58:32.533943 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:32.533894    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.594833 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.594799    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.594833 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.594832    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.595001 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.594851    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.619926 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.619891    5041 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 19:58:32.621322 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.621303    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 19:58:32.621399 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.621338    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 19:58:32.621399 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.621350    5041 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-175.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 19:58:32.621488 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.621402    5041 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.634552 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.634534    5041 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.634642 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.634592    5041 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695091 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695177 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695097    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695177 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695115    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695254 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695181    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695254 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695230    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/02aa65d6c5643e7fc09a137b81d3ea6b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"02aa65d6c5643e7fc09a137b81d3ea6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:32.695310 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:32.695277    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5bb0f42cc122a3cebecd048bef257a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" (UID: \"c5bb0f42cc122a3cebecd048bef257a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:33.370322 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.370290    5041 apiserver.go:52] "Watching apiserver"
Nov 01 19:58:33.413005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.412976    5041 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 19:58:33.413687 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.413662    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-drpqc","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q","openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj","openshift-dns/node-resolver-q6p64","openshift-multus/multus-zzrgf","openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5","openshift-ovn-kubernetes/ovnkube-node-789kh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal","openshift-insights/periodic-gathering-t2q78-k5stm","kube-system/kas-connection-checker-685895c7c9-5kdsg","openshift-image-registry/node-ca-szb2b","openshift-ingress-canary/ingress-canary-74zbl","openshift-insights/insights-operator-7c4986d449-4rbdb","openshift-network-console/networking-console-plugin-86c478bb88-hr6lk","kube-system/kas-connection-checker-685895c7c9-ppbp7","openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq","kube-system/kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw","openshift-insights/insights-runtime-extractor-922ms","openshift-console-operator/console-operator-7cd56b8dbc-pk8qq","kube-system/global-pull-secret-syncer-d549b","openshift-network-operator/iptables-alerter-q4k9b","openshift-dns/dns-default-6svs9","openshift-service-ca/service-ca-5fffc97bdf-62pdr","openshift-multus/multus-additional-cni-plugins-5d5kw","openshift-ingress/router-default-846f8d77f5-km598","openshift-multus/network-metrics-daemon-244s9","openshift-image-registry/image-registry-77f5b9cf45-wqbcb","openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w","openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg","kube-system/konnectivity-agent-qdn6f","openshift-network-diagnostics/network-check-target-j5t7b","openshift-image-registry/image-registry-59b96df89b-pmb2t","kube-system/kas-connection-checker-685895c7c9-8n2gw"]
Nov 01 19:58:33.415029 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.415008    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.416306 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.416283    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.416603 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.416582    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.417771 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.417753    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.418317 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.418297    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.418411 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.418376    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.418621 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.418603    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:33.420305 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.420286    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.421315 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.421292    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 19:58:33.421579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.421549    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 19:58:33.421877 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.421859    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 19:58:33.422083 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.422062    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 19:58:33.422313 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.422295    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 19:58:33.422409 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.422391    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.422737 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.422717    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 19:58:33.423072 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.422825    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.423423 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423403    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 19:58:33.423549 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423527    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 19:58:33.423608 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423559    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 19:58:33.423743 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423723    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 19:58:33.423838 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423822    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.423926 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423913    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.424027 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424007    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.424140 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424047    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.424199 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424184    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 19:58:33.424267 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424249    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.424518 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424500    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.424617 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424599    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 19:58:33.424676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.423409    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.424729 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424570    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.424821 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424804    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 19:58:33.425365 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.424569    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-n856f\""
Nov 01 19:58:33.426219 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426197    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.426312 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426258    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 19:58:33.426312 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426275    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.426312 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426275    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.426312 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426303    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426339    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426305    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426358    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426338    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426374    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 19:58:33.426513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426464    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.426723 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426592    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 19:58:33.426769 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426721    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 19:58:33.426961 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.426941    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.427222 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427206    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 19:58:33.427299 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427283    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 19:58:33.427384 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427368    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 19:58:33.427417 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427358    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 19:58:33.427459 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427427    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.427611 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427596    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.427969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.427947    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\""
Nov 01 19:58:33.428812 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.428790    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 19:58:33.429421 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.429400    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 19:58:33.430461 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.430423    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.430616 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.430602    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.430672 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.430621    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j2hms\""
Nov 01 19:58:33.430719 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.430694    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\""
Nov 01 19:58:33.431731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.431714    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 19:58:33.431978 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.431963    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 19:58:33.432073 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432012    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.432073 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432016    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 19:58:33.432073 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432016    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\""
Nov 01 19:58:33.432355 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432076    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\""
Nov 01 19:58:33.432355 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432126    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.432355 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432168    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\""
Nov 01 19:58:33.432355 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432183    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 19:58:33.432355 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432216    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432367    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432528    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432538    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432550    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432582    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432603    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\""
Nov 01 19:58:33.432624 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432613    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\""
Nov 01 19:58:33.432952 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432689    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 19:58:33.432952 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432723    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.432952 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432763    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.432952 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.432895    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\""
Nov 01 19:58:33.433115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.433094    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 19:58:33.433634 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.433615    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.433723 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.433672    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.433723 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.433700    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.434043 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434016    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\""
Nov 01 19:58:33.434228 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434208    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 19:58:33.434562 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434549    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 19:58:33.434652 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434635    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.434728 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434706    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\""
Nov 01 19:58:33.434918 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.434905    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\""
Nov 01 19:58:33.435532 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.435515    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 19:58:33.435794 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.435760    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.435853 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.435800    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 19:58:33.435853 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.435849    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 19:58:33.436650 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.436629    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\""
Nov 01 19:58:33.436716 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.436677    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.436716 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.436701    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\""
Nov 01 19:58:33.436841 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.436823    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 19:58:33.436933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.436916    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 19:58:33.437747 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.437723    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.437972 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.437953    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 19:58:33.438251 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.438234    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.438392 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.438375    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\""
Nov 01 19:58:33.438525 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.438506    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.438581 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.438530    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bgvh2\""
Nov 01 19:58:33.438581 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.438571    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.439727 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.439707    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 19:58:33.440296 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.440277    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\""
Nov 01 19:58:33.440374 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.440362    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 19:58:33.443881 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.443864    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96bd4\""
Nov 01 19:58:33.444809 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.444788    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 19:58:33.444880 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.444829    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 19:58:33.445044 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.445029    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 19:58:33.445096 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.445048    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 19:58:33.445096 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.445061    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xrvt7\""
Nov 01 19:58:33.477865 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.477839    5041 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:33.486929 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.486908    5041 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 19:58:33.487031 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.486950    5041 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-175.us-east-2.compute.internal"
Nov 01 19:58:33.488823 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.488808    5041 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 19:58:33.499807 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499768    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.499900 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499887    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.499936 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.499936 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499930    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.500041 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499950    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.500041 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499972    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z45gb\" (UniqueName: \"kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.500041 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.499997    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.500190 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500040    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9666c411-ce1d-4592-82de-51b88b02ea47-tmp-dir\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.500190 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500093    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-multus-daemon-config\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.500190 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500108    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.500190 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500165    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/46c95e40-824d-468a-99f4-57ec8b88fb42-config\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.500372 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b456599-bf71-4bab-afa7-e2978c6e4e42-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.500372 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500230    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.500372 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500268    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.500372 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500329    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.500564 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500365    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.500564 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500405    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.500564 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500453    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.500564 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500457    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.500698 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500566    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.500698 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500616    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvt6\" (UniqueName: \"kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.500698 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500645    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6533ed5f-4009-45be-8d6f-b90f1848d640-config-volume\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.500698 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500654    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.500886 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500715    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.500886 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500754    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gr9\" (UniqueName: \"kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.500886 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500802    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.500886 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.500886 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500864    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500894    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500920    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500944    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500968    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500978    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-tmp\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.500986    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501015    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501070    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.501115 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501089    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501259    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501283    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-config\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501289    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501356    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acc92340-bb89-4ff1-838e-08a6574531b9-original-pull-secret\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.501430 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501413    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-serviceca\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501449    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501476    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501506    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501578    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75gk\" (UniqueName: \"kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk\") pod \"migrator-66df5d496b-gdpfq\" (UID: \"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501607    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.501658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501630    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501690    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501668    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-key\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501715    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501825    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501873    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljsg\" (UniqueName: \"kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501945    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdz7p\" (UniqueName: \"kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.502007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.501972    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502009    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wlz\" (UniqueName: \"kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502144    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-serving-cert\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502159    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502226    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502272    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502279    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2ebd17c-0816-4e50-b17b-bb1ca3633730-nginx-conf\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502302    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkc5\" (UniqueName: \"kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5\") pod \"kas-connection-checker-685895c7c9-8n2gw\" (UID: \"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4\") " pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:33.502353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502333    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502382    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66grf\" (UniqueName: \"kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfbd1ed-3a02-4334-b370-328f2a511729-service-ca-bundle\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502455    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502487    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-agent-certs\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502541    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502553    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj62\" (UniqueName: \"kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502630    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502656    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502679    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.502731 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502733    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzbc\" (UniqueName: \"kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc\") pod \"volume-data-source-validator-58c9f5f46d-qrm2q\" (UID: \"2b1aad29-2c51-448a-92cd-abc01fe696d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502763    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502809    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502853    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx67m\" (UniqueName: \"kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502896    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-config\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502912    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502934    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.502980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503016    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503057    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503087    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503095    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-tuned\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503148    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503165    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/adfbd1ed-3a02-4334-b370-328f2a511729-snapshots\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503168    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503195    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503212    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f4b\" (UniqueName: \"kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503276    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.503308 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503307    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-env-overrides\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503357    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503364    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503401    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503432    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503505    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503531    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503552    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503588    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d887126-1d78-4f44-afaa-28b9d682e211-ovnkube-script-lib\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503600    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503667    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503673    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/adfbd1ed-3a02-4334-b370-328f2a511729-serving-cert\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503701    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503710    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c020f3c7-45a8-4226-9e41-289a4a7ab737-tmp\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503767    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d887126-1d78-4f44-afaa-28b9d682e211-ovn-node-metrics-cert\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503768    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503818    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503868    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv5d\" (UniqueName: \"kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d\") pod \"migrator-66df5d496b-5b24w\" (UID: \"5bbad68d-9021-4665-9315-d4ac8521f567\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503898    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503946    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f68e3cb-db93-4919-acb4-b60652805f67-cni-binary-copy\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503964    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.503992    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504018    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504047    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504071    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6533ed5f-4009-45be-8d6f-b90f1848d640-tmp-dir\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504092    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504120    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bgd\" (UniqueName: \"kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd\") pod \"kas-connection-checker-685895c7c9-5kdsg\" (UID: \"6fb413b1-e547-411a-a27a-50716d2bd1ef\") " pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504148    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb2f\" (UniqueName: \"kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f\") pod \"kas-connection-checker-685895c7c9-ppbp7\" (UID: \"087c078b-50e9-4942-ac02-6faf11ffecf3\") " pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504178    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504202    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504209    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-e221-4bc2-aec2-af48980dc5eb-konnectivity-ca\") pod \"konnectivity-agent-qdn6f\" (UID: \"a1c41cc7-e221-4bc2-aec2-af48980dc5eb\") " pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504205    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfpz\" (UniqueName: \"kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504252    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504277    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504288    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504312    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.504351 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504357    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504384    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504421    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504455    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgtp\" (UniqueName: \"kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504498    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544b6ff8-e957-4b48-880f-2311ad4aba68-signing-cabundle\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504535    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5lx\" (UniqueName: \"kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504554    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b456599-bf71-4bab-afa7-e2978c6e4e42-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504631    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504671    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwn8\" (UniqueName: \"kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504698    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504747    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504872    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2s7r\" (UniqueName: \"kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504904    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dg6\" (UniqueName: \"kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.504985    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505031    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg5c\" (UniqueName: \"kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505144    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505240    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505278    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505362    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505420    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-default-certificate\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505481    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505515    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505543    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505593    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bkf\" (UniqueName: \"kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505603    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cbb64076-bc8a-41d9-86b7-179dba150c84-iptables-alerter-script\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505625    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/379fc616-e382-4250-8ec7-34217917ea30-data-volume\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505647    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505688    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505772    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.505983    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.506005 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506009    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbfs\" (UniqueName: \"kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs\") pod \"network-check-source-6f78d59f99-djwlz\" (UID: \"2fad2018-8963-46b5-a399-e165e48dde59\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506033    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506100    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506182    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5hj\" (UniqueName: \"kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506210    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506215    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506243    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506271    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506364    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506412    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/46c95e40-824d-468a-99f4-57ec8b88fb42-serving-cert\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506456    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506503    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506537    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdb9\" (UniqueName: \"kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-stats-auth\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506577    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506652    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-trusted-ca\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506655    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-cert\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:33.507175 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.506695    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.509256 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvt6\" (UniqueName: \"kubernetes.io/projected/46c95e40-824d-468a-99f4-57ec8b88fb42-kube-api-access-5bvt6\") pod \"service-ca-operator-544d749c8c-grprg\" (UID: \"46c95e40-824d-468a-99f4-57ec8b88fb42\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg"
Nov 01 19:58:33.509365 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509298    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gr9\" (UniqueName: \"kubernetes.io/projected/379fc616-e382-4250-8ec7-34217917ea30-kube-api-access-n8gr9\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.509365 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509319    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75gk\" (UniqueName: \"kubernetes.io/projected/888a6838-2cc9-4bc5-9ca0-52f1f4052ef4-kube-api-access-j75gk\") pod \"migrator-66df5d496b-gdpfq\" (UID: \"888a6838-2cc9-4bc5-9ca0-52f1f4052ef4\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gdpfq"
Nov 01 19:58:33.509365 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.509465 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509364    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z45gb\" (UniqueName: \"kubernetes.io/projected/6533ed5f-4009-45be-8d6f-b90f1848d640-kube-api-access-z45gb\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.509574 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509559    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.509744 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.509725    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljsg\" (UniqueName: \"kubernetes.io/projected/adfbd1ed-3a02-4334-b370-328f2a511729-kube-api-access-8ljsg\") pod \"insights-operator-7c4986d449-4rbdb\" (UID: \"adfbd1ed-3a02-4334-b370-328f2a511729\") " pod="openshift-insights/insights-operator-7c4986d449-4rbdb"
Nov 01 19:58:33.510353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.510331    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdz7p\" (UniqueName: \"kubernetes.io/projected/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kube-api-access-mdz7p\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.510472 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.510439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj62\" (UniqueName: \"kubernetes.io/projected/544b6ff8-e957-4b48-880f-2311ad4aba68-kube-api-access-rwj62\") pod \"service-ca-5fffc97bdf-62pdr\" (UID: \"544b6ff8-e957-4b48-880f-2311ad4aba68\") " pod="openshift-service-ca/service-ca-5fffc97bdf-62pdr"
Nov 01 19:58:33.510539 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.510468    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66grf\" (UniqueName: \"kubernetes.io/projected/c14186d7-dc3e-4df1-9358-806da5cbc684-kube-api-access-66grf\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.510878 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.510861    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wlz\" (UniqueName: \"kubernetes.io/projected/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-kube-api-access-q5wlz\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.510978 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.510962    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.512641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.512617    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv5d\" (UniqueName: \"kubernetes.io/projected/5bbad68d-9021-4665-9315-d4ac8521f567-kube-api-access-chv5d\") pod \"migrator-66df5d496b-5b24w\" (UID: \"5bbad68d-9021-4665-9315-d4ac8521f567\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b24w"
Nov 01 19:58:33.512753 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.512735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzbc\" (UniqueName: \"kubernetes.io/projected/2b1aad29-2c51-448a-92cd-abc01fe696d1-kube-api-access-znzbc\") pod \"volume-data-source-validator-58c9f5f46d-qrm2q\" (UID: \"2b1aad29-2c51-448a-92cd-abc01fe696d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-qrm2q"
Nov 01 19:58:33.512955 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.512935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f4b\" (UniqueName: \"kubernetes.io/projected/91b01765-cd4d-46bb-b53e-54d52f846a57-kube-api-access-l5f4b\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.513141 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513100    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"periodic-gathering-t2q78-k5stm\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") " pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513362    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkc5\" (UniqueName: \"kubernetes.io/projected/72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4-kube-api-access-bbkc5\") pod \"kas-connection-checker-685895c7c9-8n2gw\" (UID: \"72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4\") " pod="kube-system/kas-connection-checker-685895c7c9-8n2gw"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513388    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5s2\" (UniqueName: \"kubernetes.io/projected/efd428db-ed11-4811-b94d-76b9e12e4288-kube-api-access-bj5s2\") pod \"network-check-target-j5t7b\" (UID: \"efd428db-ed11-4811-b94d-76b9e12e4288\") " pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513466    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgtp\" (UniqueName: \"kubernetes.io/projected/8961d680-2f89-4358-a9fc-ff43d01e2eda-kube-api-access-tvgtp\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513466    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2s7r\" (UniqueName: \"kubernetes.io/projected/25a1313c-bd53-44ed-8984-6fc0f890fc3c-kube-api-access-s2s7r\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513610    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb2f\" (UniqueName: \"kubernetes.io/projected/087c078b-50e9-4942-ac02-6faf11ffecf3-kube-api-access-kjb2f\") pod \"kas-connection-checker-685895c7c9-ppbp7\" (UID: \"087c078b-50e9-4942-ac02-6faf11ffecf3\") " pod="kube-system/kas-connection-checker-685895c7c9-ppbp7"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513610    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx67m\" (UniqueName: \"kubernetes.io/projected/9b456599-bf71-4bab-afa7-e2978c6e4e42-kube-api-access-bx67m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj\" (UID: \"9b456599-bf71-4bab-afa7-e2978c6e4e42\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.513752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfpz\" (UniqueName: \"kubernetes.io/projected/9666c411-ce1d-4592-82de-51b88b02ea47-kube-api-access-gnfpz\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5lx\" (UniqueName: \"kubernetes.io/projected/7f68e3cb-db93-4919-acb4-b60652805f67-kube-api-access-lk5lx\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514065    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bgd\" (UniqueName: \"kubernetes.io/projected/6fb413b1-e547-411a-a27a-50716d2bd1ef-kube-api-access-j9bgd\") pod \"kas-connection-checker-685895c7c9-5kdsg\" (UID: \"6fb413b1-e547-411a-a27a-50716d2bd1ef\") " pod="kube-system/kas-connection-checker-685895c7c9-5kdsg"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbfs\" (UniqueName: \"kubernetes.io/projected/2fad2018-8963-46b5-a399-e165e48dde59-kube-api-access-xdbfs\") pod \"network-check-source-6f78d59f99-djwlz\" (UID: \"2fad2018-8963-46b5-a399-e165e48dde59\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-djwlz"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dg6\" (UniqueName: \"kubernetes.io/projected/053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0-kube-api-access-69dg6\") pod \"console-operator-7cd56b8dbc-pk8qq\" (UID: \"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514531    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.514570 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514554    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5hj\" (UniqueName: \"kubernetes.io/projected/c020f3c7-45a8-4226-9e41-289a4a7ab737-kube-api-access-bx5hj\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.515266 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.514872    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdb9\" (UniqueName: \"kubernetes.io/projected/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-kube-api-access-pzdb9\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.515266 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.515077    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwn8\" (UniqueName: \"kubernetes.io/projected/7d887126-1d78-4f44-afaa-28b9d682e211-kube-api-access-wjwn8\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.515266 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.515188    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bkf\" (UniqueName: \"kubernetes.io/projected/cbb64076-bc8a-41d9-86b7-179dba150c84-kube-api-access-v2bkf\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.515266 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.515216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg5c\" (UniqueName: \"kubernetes.io/projected/a5aab77d-3a73-46b8-ba03-03e96ebf16f0-kube-api-access-xxg5c\") pod \"ingress-canary-74zbl\" (UID: \"a5aab77d-3a73-46b8-ba03-03e96ebf16f0\") " pod="openshift-ingress-canary/ingress-canary-74zbl"
Nov 01 19:58:33.607106 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607077    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.607106 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607109    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607126    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607171    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607192    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607214    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607237    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607242    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-bin\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607262    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607260    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-cnibin\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.607250    5041 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607297    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867-host\") pod \"node-ca-szb2b\" (UID: \"f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867\") " pod="openshift-image-registry/node-ca-szb2b"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607304    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607342 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607332    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-k8s-cni-cncf-io\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607334    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607383    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-kubelet-config\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607388    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-etc-selinux\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607404    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607359    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acc92340-bb89-4ff1-838e-08a6574531b9-dbus\") pod \"global-pull-secret-syncer-d549b\" (UID: \"acc92340-bb89-4ff1-838e-08a6574531b9\") " pod="kube-system/global-pull-secret-syncer-d549b"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.607421    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.107399285 +0000 UTC m=+1.828477946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-slash\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607285    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-systemd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607466    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607498    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.607506    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert podName:b2ebd17c-0816-4e50-b17b-bb1ca3633730 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.107484918 +0000 UTC m=+1.828563578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-hr6lk" (UID: "b2ebd17c-0816-4e50-b17b-bb1ca3633730") : secret "networking-console-plugin-cert" not found
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-system-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607544    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607564    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607581    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607611    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607625    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607647    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607661    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-node-log\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607670    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607691    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-netns\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607696    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607712    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-kubernetes\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607718    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-var-lib-kubelet\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607724    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-run-multus-certs\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607670    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cbb64076-bc8a-41d9-86b7-179dba150c84-host-slash\") pod \"iptables-alerter-q4k9b\" (UID: \"cbb64076-bc8a-41d9-86b7-179dba150c84\") " pod="openshift-network-operator/iptables-alerter-q4k9b"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607740    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607729    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-socket-dir-parent\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607718    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-sys\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607801    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-netns\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607833    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-var-lib-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607852    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607873    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607880    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607894    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607906    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-cni-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607910    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607917    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-kubelet\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607933    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607945    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-multus\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607949    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.607942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607954    5041 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607971    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-run\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.607974    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608000    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608006    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-socket-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.608013    5041 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608026    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-systemd-units\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608056    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.608072    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.108057042 +0000 UTC m=+1.829135707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608090    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-log-socket\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608124    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-lib-modules\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608199    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608220    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608241    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608263    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608298    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608301    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-host\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.608316    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608351    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/379fc616-e382-4250-8ec7-34217917ea30-crio-socket\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608353    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-etc-openvswitch\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608320    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.608384    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.108369517 +0000 UTC m=+1.829448187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608414    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608469    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608480    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-modprobe-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608497    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608536    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608568    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-registration-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608588    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-run-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608614    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608650    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608679    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608704    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-cni-netd\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608754    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608764    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608796    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608823    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608829    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-run-ovn\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608621    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-conf\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608850    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608873    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608896    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608917    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608945    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608961    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysctl-d\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608986    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d887126-1d78-4f44-afaa-28b9d682e211-host-kubelet\") pod \"ovnkube-node-789kh\" (UID: \"7d887126-1d78-4f44-afaa-28b9d682e211\") " pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.608995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.609017    5041 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609027    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-device-dir\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.609057    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.109044391 +0000 UTC m=+1.830123061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : configmap "kube-rbac-proxy" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609075    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609089    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab5534dc-85a6-4a5f-8101-7fc41ef4999e-sys-fs\") pod \"aws-ebs-csi-driver-node-jj244\" (UID: \"ab5534dc-85a6-4a5f-8101-7fc41ef4999e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jj244"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609096    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-cnibin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609104    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609128    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609158    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-host-var-lib-cni-bin\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.609164    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609131    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-hostroot\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609214    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-etc-kubernetes\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:33.609231    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:34.109215322 +0000 UTC m=+1.830293991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f68e3cb-db93-4919-acb4-b60652805f67-multus-conf-dir\") pod \"multus-zzrgf\" (UID: \"7f68e3cb-db93-4919-acb4-b60652805f67\") " pod="openshift-multus/multus-zzrgf"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609269    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-sysconfig\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609276    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c020f3c7-45a8-4226-9e41-289a4a7ab737-etc-systemd\") pod \"tuned-drpqc\" (UID: \"c020f3c7-45a8-4226-9e41-289a4a7ab737\") " pod="openshift-cluster-node-tuning-operator/tuned-drpqc"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609290    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9666c411-ce1d-4592-82de-51b88b02ea47-hosts-file\") pod \"node-resolver-q6p64\" (UID: \"9666c411-ce1d-4592-82de-51b88b02ea47\") " pod="openshift-dns/node-resolver-q6p64"
Nov 01 19:58:33.609884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.609308    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a1313c-bd53-44ed-8984-6fc0f890fc3c-system-cni-dir\") pod \"multus-additional-cni-plugins-5d5kw\" (UID: \"25a1313c-bd53-44ed-8984-6fc0f890fc3c\") " pod="openshift-multus/multus-additional-cni-plugins-5d5kw"
Nov 01 19:58:33.612261 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.610900    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c14186d7-dc3e-4df1-9358-806da5cbc684-metrics-certs\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:33.612261 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.610909    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6533ed5f-4009-45be-8d6f-b90f1848d640-metrics-tls\") pod \"dns-default-6svs9\" (UID: \"6533ed5f-4009-45be-8d6f-b90f1848d640\") " pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:33.612261 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.610953    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"image-registry-59b96df89b-pmb2t\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") " pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.612261 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.611329    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"image-registry-77f5b9cf45-wqbcb\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") " pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.612261 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.611378    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8961d680-2f89-4358-a9fc-ff43d01e2eda-metrics-certs\") pod \"network-metrics-daemon-244s9\" (UID: \"8961d680-2f89-4358-a9fc-ff43d01e2eda\") " pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.715400 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.715320    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:33.716186 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.716164    5041 scope.go:122] "RemoveContainer" containerID="cdcaea1b41e414a7d1dd2ff9fb5b4d77323d74f69323ddd6075bff6f61577bcb"
Nov 01 19:58:33.725986 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.725966    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:33.732346 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.732329    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-244s9"
Nov 01 19:58:33.734495 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:33.734475    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:34.032335 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.032245    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:58:34.035091 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:34.035061    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8da3aea_f54a_45ad_b9bf_4a8108f7951e.slice/crio-1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6 WatchSource:0}: Error finding container 1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6: Status 404 returned error can't find the container with id 1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6
Nov 01 19:58:34.076630 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.076608    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:58:34.079986 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:34.079963    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60703b7e_ae49_4bec_923c_73ade0101e57.slice/crio-2ab65a94343bd84b16add892f51acefb9d547abcd1913582ddfc7c02667c340c WatchSource:0}: Error finding container 2ab65a94343bd84b16add892f51acefb9d547abcd1913582ddfc7c02667c340c: Status 404 returned error can't find the container with id 2ab65a94343bd84b16add892f51acefb9d547abcd1913582ddfc7c02667c340c
Nov 01 19:58:34.114098 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114054    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:34.114198 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114134    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:34.114276 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114194    5041 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:34.114276 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114257    5041 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:34.114460 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114263    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.114243479 +0000 UTC m=+2.835322147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:34.114460 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114324    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls podName:a7cde0f8-1666-42a5-84e1-34f1ab468d1e nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.114311358 +0000 UTC m=+2.835390020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-x9tc5" (UID: "a7cde0f8-1666-42a5-84e1-34f1ab468d1e") : secret "cluster-monitoring-operator-tls" not found
Nov 01 19:58:34.114460 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:34.114623 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114518    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:34.114623 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:34.114623 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:34.114735 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.114716    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/379fc616-e382-4250-8ec7-34217917ea30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:34.114774 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114744    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.114729169 +0000 UTC m=+2.835807824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:34.114857 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114842    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:34.114910 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:34.114895    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.114881657 +0000 UTC m=+2.835960328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:34.117995 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.117598    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2ebd17c-0816-4e50-b17b-bb1ca3633730-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-hr6lk\" (UID: \"b2ebd17c-0816-4e50-b17b-bb1ca3633730\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:34.119228 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.119201    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6svs9"]
Nov 01 19:58:34.122425 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:34.122050    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6533ed5f_4009_45be_8d6f_b90f1848d640.slice/crio-b0d661823c3372421c1fc0685488bfa6fd8a12c60ecde37b689956696ee9af09 WatchSource:0}: Error finding container b0d661823c3372421c1fc0685488bfa6fd8a12c60ecde37b689956696ee9af09: Status 404 returned error can't find the container with id b0d661823c3372421c1fc0685488bfa6fd8a12c60ecde37b689956696ee9af09
Nov 01 19:58:34.124625 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.124612    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 19:58:34.155927 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.155892    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-244s9"]
Nov 01 19:58:34.160114 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:34.160085    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8961d680_2f89_4358_a9fc_ff43d01e2eda.slice/crio-8fc237390533b2f36645a9388179a72b82c014dab7a7933c0f0794ee677f54fb WatchSource:0}: Error finding container 8fc237390533b2f36645a9388179a72b82c014dab7a7933c0f0794ee677f54fb: Status 404 returned error can't find the container with id 8fc237390533b2f36645a9388179a72b82c014dab7a7933c0f0794ee677f54fb
Nov 01 19:58:34.319646 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.319609    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"
Nov 01 19:58:34.483466 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.483437    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 19:58:34.484009 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.483519    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq" event={"ID":"053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0","Type":"ContainerStarted","Data":"8b0de85d1863aa021086d274ef7eb025ed3a8d22608b730157cbc9a081c226e4"}
Nov 01 19:58:34.485374 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.485341    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" event={"ID":"60703b7e-ae49-4bec-923c-73ade0101e57","Type":"ContainerStarted","Data":"f5e1113216c2068efe3417ff2d4d7d8e5c893bc8d56381d80c18c7dedc7dcd1d"}
Nov 01 19:58:34.485499 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.485399    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" event={"ID":"60703b7e-ae49-4bec-923c-73ade0101e57","Type":"ContainerStarted","Data":"2ab65a94343bd84b16add892f51acefb9d547abcd1913582ddfc7c02667c340c"}
Nov 01 19:58:34.486602 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.486570    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6svs9" event={"ID":"6533ed5f-4009-45be-8d6f-b90f1848d640","Type":"ContainerStarted","Data":"b0d661823c3372421c1fc0685488bfa6fd8a12c60ecde37b689956696ee9af09"}
Nov 01 19:58:34.488247 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.488225    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" event={"ID":"c8da3aea-f54a-45ad-b9bf-4a8108f7951e","Type":"ContainerStarted","Data":"f8e6d3e7ea03c51c76fc2b7511908030e4cf8776eb315f2f97917b69817b4df2"}
Nov 01 19:58:34.488341 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.488265    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" event={"ID":"c8da3aea-f54a-45ad-b9bf-4a8108f7951e","Type":"ContainerStarted","Data":"1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6"}
Nov 01 19:58:34.489534 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.489514    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-244s9" event={"ID":"8961d680-2f89-4358-a9fc-ff43d01e2eda","Type":"ContainerStarted","Data":"8fc237390533b2f36645a9388179a72b82c014dab7a7933c0f0794ee677f54fb"}
Nov 01 19:58:34.604180 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:34.604103    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-hr6lk"]
Nov 01 19:58:34.607921 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:34.607893    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ebd17c_0816_4e50_b17b_bb1ca3633730.slice/crio-cc53de1c79e39d45ceecb24cd7b24a13b46699bd71c6ead994a60dd6197ccae5 WatchSource:0}: Error finding container cc53de1c79e39d45ceecb24cd7b24a13b46699bd71c6ead994a60dd6197ccae5: Status 404 returned error can't find the container with id cc53de1c79e39d45ceecb24cd7b24a13b46699bd71c6ead994a60dd6197ccae5
Nov 01 19:58:35.126189 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.126150    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:35.126469 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.126205    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:35.126469 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.126251    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:35.126469 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.126287    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:35.127615 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:35.126890    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:37.126866676 +0000 UTC m=+4.847945347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:35.127615 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:35.127292    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:35.127615 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:35.127378    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:37.127362448 +0000 UTC m=+4.848441115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:35.127615 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:35.127502    5041 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 19:58:35.127615 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:35.127541    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls podName:91b01765-cd4d-46bb-b53e-54d52f846a57 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:37.127528247 +0000 UTC m=+4.848606914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-4b8qw" (UID: "91b01765-cd4d-46bb-b53e-54d52f846a57") : secret "samples-operator-tls" not found
Nov 01 19:58:35.130249 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.130227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7cde0f8-1666-42a5-84e1-34f1ab468d1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-x9tc5\" (UID: \"a7cde0f8-1666-42a5-84e1-34f1ab468d1e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:35.216862 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.216824    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"
Nov 01 19:58:35.497507 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.496823    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk" event={"ID":"b2ebd17c-0816-4e50-b17b-bb1ca3633730","Type":"ContainerStarted","Data":"cc53de1c79e39d45ceecb24cd7b24a13b46699bd71c6ead994a60dd6197ccae5"}
Nov 01 19:58:35.497507 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.496870    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:35.497507 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.496834    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:35.513152 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:35.513088    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5"]
Nov 01 19:58:35.516435 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:35.516402    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cde0f8_1666_42a5_84e1_34f1ab468d1e.slice/crio-ce2fdc59b175079e6d2b9e3adf3bedb7f3d7796c9a09a3a5df2880f468044451 WatchSource:0}: Error finding container ce2fdc59b175079e6d2b9e3adf3bedb7f3d7796c9a09a3a5df2880f468044451: Status 404 returned error can't find the container with id ce2fdc59b175079e6d2b9e3adf3bedb7f3d7796c9a09a3a5df2880f468044451
Nov 01 19:58:36.293014 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.292957    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:36.503019 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.502979    5041 generic.go:356] "Generic (PLEG): container finished" podID="9b456599-bf71-4bab-afa7-e2978c6e4e42" containerID="b34162eb7a6cb25d1bb71c96bdec1a8e7994a8b8e9a4ed4bee7ea279e1080848" exitCode=0
Nov 01 19:58:36.503434 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.503043    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj" event={"ID":"9b456599-bf71-4bab-afa7-e2978c6e4e42","Type":"ContainerDied","Data":"b34162eb7a6cb25d1bb71c96bdec1a8e7994a8b8e9a4ed4bee7ea279e1080848"}
Nov 01 19:58:36.503434 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.503426    5041 scope.go:122] "RemoveContainer" containerID="b34162eb7a6cb25d1bb71c96bdec1a8e7994a8b8e9a4ed4bee7ea279e1080848"
Nov 01 19:58:36.505081 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.505059    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-hr6lk" event={"ID":"b2ebd17c-0816-4e50-b17b-bb1ca3633730","Type":"ContainerStarted","Data":"9fd0a3f8ad2475a979583ccf36c4113bbe96e51f032c0cfaaa498e720715ab73"}
Nov 01 19:58:36.508272 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.508248    5041 generic.go:356] "Generic (PLEG): container finished" podID="46c95e40-824d-468a-99f4-57ec8b88fb42" containerID="f018a66de2ac8f647473e5032475e974c3ace75265d476e8e99971ed01dace97" exitCode=0
Nov 01 19:58:36.508381 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.508318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg" event={"ID":"46c95e40-824d-468a-99f4-57ec8b88fb42","Type":"ContainerDied","Data":"f018a66de2ac8f647473e5032475e974c3ace75265d476e8e99971ed01dace97"}
Nov 01 19:58:36.508654 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.508633    5041 scope.go:122] "RemoveContainer" containerID="f018a66de2ac8f647473e5032475e974c3ace75265d476e8e99971ed01dace97"
Nov 01 19:58:36.510269 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.510245    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6svs9" event={"ID":"6533ed5f-4009-45be-8d6f-b90f1848d640","Type":"ContainerStarted","Data":"04fe332680e0592b9ebe4b68874ab0558a5b6869eeaf725c9dbb6e1ff65bc272"}
Nov 01 19:58:36.510377 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.510291    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6svs9" event={"ID":"6533ed5f-4009-45be-8d6f-b90f1848d640","Type":"ContainerStarted","Data":"34fa2382e566d79e75d0537d9c26a5feb8ad0380f47ec53eaf4170ed445e44e1"}
Nov 01 19:58:36.510377 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.510347    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:36.512416 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.512331    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-244s9" event={"ID":"8961d680-2f89-4358-a9fc-ff43d01e2eda","Type":"ContainerStarted","Data":"879f132c63812fd469ee851cff81b799b442fd9a3a80239668e30c697c6b2d2a"}
Nov 01 19:58:36.512416 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.512372    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-244s9" event={"ID":"8961d680-2f89-4358-a9fc-ff43d01e2eda","Type":"ContainerStarted","Data":"81aa157dbd8991e560b36bd062538dddf476ac0076c08c676e7554d4101e2a18"}
Nov 01 19:58:36.514755 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.514737    5041 generic.go:356] "Generic (PLEG): container finished" podID="adfbd1ed-3a02-4334-b370-328f2a511729" containerID="c0cfd4fd8aced840fbb1071afcf11574419b4effb12a00fccb6737bd7e18953b" exitCode=0
Nov 01 19:58:36.514877 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.514810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerDied","Data":"c0cfd4fd8aced840fbb1071afcf11574419b4effb12a00fccb6737bd7e18953b"}
Nov 01 19:58:36.515129 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.515112    5041 scope.go:122] "RemoveContainer" containerID="c0cfd4fd8aced840fbb1071afcf11574419b4effb12a00fccb6737bd7e18953b"
Nov 01 19:58:36.516923 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:36.516899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5" event={"ID":"a7cde0f8-1666-42a5-84e1-34f1ab468d1e","Type":"ContainerStarted","Data":"ce2fdc59b175079e6d2b9e3adf3bedb7f3d7796c9a09a3a5df2880f468044451"}
Nov 01 19:58:37.149670 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.149627    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:37.149884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.149722    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:37.149884 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.149756    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:37.150005 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:37.149877    5041 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:37.150005 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:37.149929    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle podName:c14186d7-dc3e-4df1-9358-806da5cbc684 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:41.149911019 +0000 UTC m=+8.870989693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle") pod "router-default-846f8d77f5-km598" (UID: "c14186d7-dc3e-4df1-9358-806da5cbc684") : configmap references non-existent config key: service-ca.crt
Nov 01 19:58:37.150005 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:37.149948    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls podName:379fc616-e382-4250-8ec7-34217917ea30 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:41.149939634 +0000 UTC m=+8.871018290 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls") pod "insights-runtime-extractor-922ms" (UID: "379fc616-e382-4250-8ec7-34217917ea30") : secret "insights-runtime-extractor-tls" not found
Nov 01 19:58:37.155561 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.155533    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b01765-cd4d-46bb-b53e-54d52f846a57-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-4b8qw\" (UID: \"91b01765-cd4d-46bb-b53e-54d52f846a57\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:37.317630 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.317592    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"
Nov 01 19:58:37.407348 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.407287    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:58:37.522447 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.522419    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-hlqlj" event={"ID":"9b456599-bf71-4bab-afa7-e2978c6e4e42","Type":"ContainerStarted","Data":"f1f5c721cb24aeb4f06e481df1c3a87089d0738bec50200f54096797b86dbe3f"}
Nov 01 19:58:37.524126 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.524104    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-grprg" event={"ID":"46c95e40-824d-468a-99f4-57ec8b88fb42","Type":"ContainerStarted","Data":"b6561c3c364e360c9c2675468aa852bb43cc34d34ec74fe3115746f7a72f7529"}
Nov 01 19:58:37.525876 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.525848    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerStarted","Data":"a52765ad9f86762dd01a5039ff73b6a8f690c2c6a14b5f52dbc3e0ce58c0133f"}
Nov 01 19:58:37.599967 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:37.599928    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw"]
Nov 01 19:58:38.532746 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:38.532697    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-x9tc5" event={"ID":"a7cde0f8-1666-42a5-84e1-34f1ab468d1e","Type":"ContainerStarted","Data":"86931247fdd4da512333967156f3d42ebd4188263a1b3791aa74e619082dff72"}
Nov 01 19:58:38.534034 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:38.534006    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw" event={"ID":"91b01765-cd4d-46bb-b53e-54d52f846a57","Type":"ContainerStarted","Data":"4e4fe6b9b4c4886a9c25483d8c89252609063077ba919871efa978aee6b58b65"}
Nov 01 19:58:39.540051 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:39.540013    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw" event={"ID":"91b01765-cd4d-46bb-b53e-54d52f846a57","Type":"ContainerStarted","Data":"0a43a0005c769e5335f330e356cf31a9d4485c140ca169c14d3fc21cf9310ceb"}
Nov 01 19:58:39.540507 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:39.540103    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-4b8qw" event={"ID":"91b01765-cd4d-46bb-b53e-54d52f846a57","Type":"ContainerStarted","Data":"4ff7159e1fcc72a31430cec930d5f9d8a705af34a6617bdd47a3e6a8063ade4c"}
Nov 01 19:58:40.069651 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:40.069616    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:40.461307 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:40.461229    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:40.461489 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:40.461465    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:41.187207 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.187164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:41.187590 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.187213    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:41.187758 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.187739    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c14186d7-dc3e-4df1-9358-806da5cbc684-service-ca-bundle\") pod \"router-default-846f8d77f5-km598\" (UID: \"c14186d7-dc3e-4df1-9358-806da5cbc684\") " pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:41.189460 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.189439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/379fc616-e382-4250-8ec7-34217917ea30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-922ms\" (UID: \"379fc616-e382-4250-8ec7-34217917ea30\") " pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:41.218454 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.218427    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:41.237174 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.237114    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-922ms"
Nov 01 19:58:41.495035 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.494854    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-846f8d77f5-km598"]
Nov 01 19:58:41.497711 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:41.497682    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14186d7_dc3e_4df1_9358_806da5cbc684.slice/crio-22a991efab8fd444d804b3d7ec4fa79cd88b9b06d47d0138e37212ee105b907f WatchSource:0}: Error finding container 22a991efab8fd444d804b3d7ec4fa79cd88b9b06d47d0138e37212ee105b907f: Status 404 returned error can't find the container with id 22a991efab8fd444d804b3d7ec4fa79cd88b9b06d47d0138e37212ee105b907f
Nov 01 19:58:41.515747 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.515725    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-922ms"]
Nov 01 19:58:41.529824 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:41.529801    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379fc616_e382_4250_8ec7_34217917ea30.slice/crio-af116b6cf39ea1ca8d3aa27a43f785efad398c0dcbe090c150ee16bec829590f WatchSource:0}: Error finding container af116b6cf39ea1ca8d3aa27a43f785efad398c0dcbe090c150ee16bec829590f: Status 404 returned error can't find the container with id af116b6cf39ea1ca8d3aa27a43f785efad398c0dcbe090c150ee16bec829590f
Nov 01 19:58:41.549814 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.549768    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-922ms" event={"ID":"379fc616-e382-4250-8ec7-34217917ea30","Type":"ContainerStarted","Data":"af116b6cf39ea1ca8d3aa27a43f785efad398c0dcbe090c150ee16bec829590f"}
Nov 01 19:58:41.551502 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:41.551475    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-846f8d77f5-km598" event={"ID":"c14186d7-dc3e-4df1-9358-806da5cbc684","Type":"ContainerStarted","Data":"22a991efab8fd444d804b3d7ec4fa79cd88b9b06d47d0138e37212ee105b907f"}
Nov 01 19:58:42.088591 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.088558    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:58:42.313967 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.313929    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:42.314380 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.314077    5041 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 19:58:42.318672 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.318646    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-pk8qq"
Nov 01 19:58:42.557330 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.557296    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-922ms" event={"ID":"379fc616-e382-4250-8ec7-34217917ea30","Type":"ContainerStarted","Data":"5ef4383d3f950ee28fc3ccb5f3137349ee53b2ab2f00a1a2100cb7d0ce34b6c3"}
Nov 01 19:58:42.557517 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.557347    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-922ms" event={"ID":"379fc616-e382-4250-8ec7-34217917ea30","Type":"ContainerStarted","Data":"8148cda5eca5c73d6729051763e1d1193a8ad727a4ed9da468b786f395dd65e9"}
Nov 01 19:58:42.558790 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:42.558759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-846f8d77f5-km598" event={"ID":"c14186d7-dc3e-4df1-9358-806da5cbc684","Type":"ContainerStarted","Data":"fd5654cf6d623c662d39df8b6d84a24674e982e54e7218b732e327a877d201d9"}
Nov 01 19:58:43.219289 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:43.219252    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:43.222295 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:43.222270    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:44.064679 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:44.064649    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:44.065942 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:44.065923    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-846f8d77f5-km598"
Nov 01 19:58:44.570905 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:44.570868    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-922ms" event={"ID":"379fc616-e382-4250-8ec7-34217917ea30","Type":"ContainerStarted","Data":"3a79b3e48314a1aaccebaa4035ae27e14c5b4685f5066f56000268474c00e3c7"}
Nov 01 19:58:45.706191 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:45.706151    5041 patch_prober.go:29] interesting pod/image-registry-59b96df89b-pmb2t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 19:58:45.706562 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:45.706217    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 19:58:46.529071 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:46.529040    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6svs9"
Nov 01 19:58:46.658007 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:46.657966    5041 patch_prober.go:29] interesting pod/image-registry-77f5b9cf45-wqbcb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 19:58:46.658167 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:46.658025    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 19:58:49.372271 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.372233    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:58:49.377595 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.377568    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.394361 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.394333    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:58:49.395220 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.395198    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6bdrz"]
Nov 01 19:58:49.407169 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.407123    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"]
Nov 01 19:58:49.408578 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.408555    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.410605 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.410584    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:58:49.410851 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.410828    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:49.414097 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.414071    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.417306 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.417112    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:58:49.423044 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.422711    5041 patch_prober.go:29] interesting pod/image-registry-77f5b9cf45-wqbcb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 19:58:49.423044 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.422764    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 19:58:49.423044 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.422991    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"]
Nov 01 19:58:49.424312 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424292    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 19:58:49.424411 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424374    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-skz89\""
Nov 01 19:58:49.424411 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424400    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 19:58:49.424804 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424442    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 19:58:49.424804 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424518    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-snwvx\""
Nov 01 19:58:49.424804 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424530    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wgcdr\""
Nov 01 19:58:49.424804 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.424604    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 19:58:49.426676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.426272    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:58:49.426676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.426535    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 19:58:49.426676 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.426652    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 19:58:49.427256 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.427235    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6bdrz"]
Nov 01 19:58:49.433386 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.433290    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 19:58:49.433386 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.433380    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 19:58:49.471792 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471756    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.471928 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471848    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcwpt\" (UniqueName: \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.471928 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471875    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.472000 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471938    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnbk\" (UniqueName: \"kubernetes.io/projected/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-kube-api-access-qcnbk\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.472000 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471972    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-tmp\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.472000 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.471997    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.491363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.491324    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bb76fff67-jsvjg"]
Nov 01 19:58:49.494843 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.494824    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.507177 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.507155    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bb76fff67-jsvjg"]
Nov 01 19:58:49.572906 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.572875    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bbc2e4-cb60-4916-ad80-b38466c75efd-ca-trust-extracted\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573104 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.572913    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573104 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.572929    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-installation-pull-secrets\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573104 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573010    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573104 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.573104 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573094    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9752474-48e4-4bad-bc8a-205de8b30144-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nk7cf\" (UID: \"d9752474-48e4-4bad-bc8a-205de8b30144\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573123    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-tls\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573150    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573204    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573231    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-image-registry-private-configuration\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573247    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573263    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-certificates\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573297 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573282    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573554 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573324    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-trusted-ca\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573554 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573405    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcwpt\" (UniqueName: \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.573641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573548    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.573641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573623    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdlc\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-kube-api-access-pfdlc\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573735 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573660    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnbk\" (UniqueName: \"kubernetes.io/projected/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-kube-api-access-qcnbk\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.573735 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573702    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-bound-sa-token\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.573866 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573738    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-tmp\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.573866 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573742    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.573866 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573824    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzx8\" (UniqueName: \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.573866 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.573849    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.574123 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.574101    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-tmp\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.574226 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.574188    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.574545 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.574523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.581872 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.581845    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcwpt\" (UniqueName: \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\") pod \"periodic-gathering-hclz8-58g2v\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") " pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.581999 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.581982    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnbk\" (UniqueName: \"kubernetes.io/projected/f631778e-0855-4a7f-8e2d-6c0385f9bbbe-kube-api-access-qcnbk\") pod \"downloads-6dd789c8b7-6bdrz\" (UID: \"f631778e-0855-4a7f-8e2d-6c0385f9bbbe\") " pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.674638 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674559    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzx8\" (UniqueName: \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.674638 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674605    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bbc2e4-cb60-4916-ad80-b38466c75efd-ca-trust-extracted\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.674638 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674622    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674732    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-installation-pull-secrets\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674772    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674824    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9752474-48e4-4bad-bc8a-205de8b30144-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nk7cf\" (UID: \"d9752474-48e4-4bad-bc8a-205de8b30144\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674844    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-tls\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674872    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.674913 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674899    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674938    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-image-registry-private-configuration\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674959    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.674981    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-certificates\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675010    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bbc2e4-cb60-4916-ad80-b38466c75efd-ca-trust-extracted\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675072    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-trusted-ca\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdlc\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-kube-api-access-pfdlc\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.675210 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675184    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-bound-sa-token\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.676725 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675753    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.676725 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675894    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-certificates\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.676725 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.675918    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.676725 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.676680    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.677085 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.676903    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bbc2e4-cb60-4916-ad80-b38466c75efd-trusted-ca\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.677362 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.677338    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.677944 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.677917    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-installation-pull-secrets\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.678418 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.678388    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.678513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.678485    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bbc2e4-cb60-4916-ad80-b38466c75efd-image-registry-private-configuration\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.678513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.678501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-registry-tls\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.678630 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.678503    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d9752474-48e4-4bad-bc8a-205de8b30144-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-nk7cf\" (UID: \"d9752474-48e4-4bad-bc8a-205de8b30144\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:49.678852 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.678830    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.683091 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.683066    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdlc\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-kube-api-access-pfdlc\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.683679 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.683655    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bbc2e4-cb60-4916-ad80-b38466c75efd-bound-sa-token\") pod \"image-registry-bb76fff67-jsvjg\" (UID: \"03bbc2e4-cb60-4916-ad80-b38466c75efd\") " pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.683773 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.683756    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzx8\" (UniqueName: \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\") pod \"console-dc47fccc9-z2fz4\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") " pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.706579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.706550    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:58:49.734881 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.734854    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:58:49.747122 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.746958    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:49.752107 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.752084    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:49.807991 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.807461    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:49.979155 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:49.979026    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:58:49.983901 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:49.983082    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552dbe68_8fe8_4a86_b74a_49d441b5075a.slice/crio-3b305d68c47ad2f4288fd4b61c3edc98176216901b1e5782ce1fe1f032f4844e WatchSource:0}: Error finding container 3b305d68c47ad2f4288fd4b61c3edc98176216901b1e5782ce1fe1f032f4844e: Status 404 returned error can't find the container with id 3b305d68c47ad2f4288fd4b61c3edc98176216901b1e5782ce1fe1f032f4844e
Nov 01 19:58:50.023825 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.023797    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6bdrz"]
Nov 01 19:58:50.026523 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:50.026500    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf631778e_0855_4a7f_8e2d_6c0385f9bbbe.slice/crio-b93ab4656f26ddd75457d025fbb0295e653afe786271a806d9a909c56f4a620e WatchSource:0}: Error finding container b93ab4656f26ddd75457d025fbb0295e653afe786271a806d9a909c56f4a620e: Status 404 returned error can't find the container with id b93ab4656f26ddd75457d025fbb0295e653afe786271a806d9a909c56f4a620e
Nov 01 19:58:50.054608 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.054581    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"]
Nov 01 19:58:50.055613 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.055594    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:58:50.057910 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:50.057881    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418a4dcb_fb3f_4917_9da9_cf43e683e335.slice/crio-a209ffce8a488691f5d9c55de4adc4abc95fa6f87bf9c6d1a9b0eaa165d91a26 WatchSource:0}: Error finding container a209ffce8a488691f5d9c55de4adc4abc95fa6f87bf9c6d1a9b0eaa165d91a26: Status 404 returned error can't find the container with id a209ffce8a488691f5d9c55de4adc4abc95fa6f87bf9c6d1a9b0eaa165d91a26
Nov 01 19:58:50.064421 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:50.064394    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9752474_48e4_4bad_bc8a_205de8b30144.slice/crio-b1ac551815099ba86825a5a4e58d58ba577ed5f6196a032eddaa3d57885ba2df WatchSource:0}: Error finding container b1ac551815099ba86825a5a4e58d58ba577ed5f6196a032eddaa3d57885ba2df: Status 404 returned error can't find the container with id b1ac551815099ba86825a5a4e58d58ba577ed5f6196a032eddaa3d57885ba2df
Nov 01 19:58:50.116049 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.116018    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bb76fff67-jsvjg"]
Nov 01 19:58:50.120034 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:50.120009    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bbc2e4_cb60_4916_ad80_b38466c75efd.slice/crio-af0028a7f053903a040cd9b4026992249f427176a39f749ca967b9305d35075d WatchSource:0}: Error finding container af0028a7f053903a040cd9b4026992249f427176a39f749ca967b9305d35075d: Status 404 returned error can't find the container with id af0028a7f053903a040cd9b4026992249f427176a39f749ca967b9305d35075d
Nov 01 19:58:50.610191 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.610159    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg" event={"ID":"03bbc2e4-cb60-4916-ad80-b38466c75efd","Type":"ContainerStarted","Data":"bc2fe70941b8305b6101d3a6739c8ebff9b8b122a3ba0fbcf37b57f3bab90bbc"}
Nov 01 19:58:50.610641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.610206    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg" event={"ID":"03bbc2e4-cb60-4916-ad80-b38466c75efd","Type":"ContainerStarted","Data":"af0028a7f053903a040cd9b4026992249f427176a39f749ca967b9305d35075d"}
Nov 01 19:58:50.610641 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.610263    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:58:50.611299 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.611271    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf" event={"ID":"d9752474-48e4-4bad-bc8a-205de8b30144","Type":"ContainerStarted","Data":"b1ac551815099ba86825a5a4e58d58ba577ed5f6196a032eddaa3d57885ba2df"}
Nov 01 19:58:50.612845 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.612820    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hclz8-58g2v" event={"ID":"552dbe68-8fe8-4a86-b74a-49d441b5075a","Type":"ContainerStarted","Data":"daf4cc2c598e45427e22489451814d67b1b4e68954f7b9eb69ff6bd3f454f7c3"}
Nov 01 19:58:50.612968 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.612878    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hclz8-58g2v" event={"ID":"552dbe68-8fe8-4a86-b74a-49d441b5075a","Type":"ContainerStarted","Data":"3b305d68c47ad2f4288fd4b61c3edc98176216901b1e5782ce1fe1f032f4844e"}
Nov 01 19:58:50.614093 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.614070    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc47fccc9-z2fz4" event={"ID":"418a4dcb-fb3f-4917-9da9-cf43e683e335","Type":"ContainerStarted","Data":"a209ffce8a488691f5d9c55de4adc4abc95fa6f87bf9c6d1a9b0eaa165d91a26"}
Nov 01 19:58:50.615217 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.615193    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6bdrz" event={"ID":"f631778e-0855-4a7f-8e2d-6c0385f9bbbe","Type":"ContainerStarted","Data":"b93ab4656f26ddd75457d025fbb0295e653afe786271a806d9a909c56f4a620e"}
Nov 01 19:58:50.627871 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.627809    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg" podStartSLOduration=1.627766223 podStartE2EDuration="1.627766223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:50.627617539 +0000 UTC m=+18.348696215" watchObservedRunningTime="2026-11-01 19:58:50.627766223 +0000 UTC m=+18.348844900"
Nov 01 19:58:50.644317 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:50.644273    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-hclz8-58g2v" podStartSLOduration=1.644261261 podStartE2EDuration="1.644261261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:50.643157148 +0000 UTC m=+18.364235825" watchObservedRunningTime="2026-11-01 19:58:50.644261261 +0000 UTC m=+18.365339936"
Nov 01 19:58:51.624418 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:51.624379    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf" event={"ID":"d9752474-48e4-4bad-bc8a-205de8b30144","Type":"ContainerStarted","Data":"62f551b666c332d510ff371e23406f8e01f531531806ef57d8384cb351209540"}
Nov 01 19:58:51.625176 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:51.625152    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:51.631385 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:51.631363    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf"
Nov 01 19:58:51.639428 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:51.639373    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-nk7cf" podStartSLOduration=1.28741448 podStartE2EDuration="2.639358561s" totalImagesPullingTime="1.351944081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:51.638412938 +0000 UTC m=+19.359491615" watchObservedRunningTime="2026-11-01 19:58:51.639358561 +0000 UTC m=+19.360437239"
Nov 01 19:58:52.093684 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:52.093653    5041 patch_prober.go:29] interesting pod/image-registry-59b96df89b-pmb2t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 19:58:52.093883 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:52.093709    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 19:58:53.366181 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:53.366143    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6svs9_6533ed5f-4009-45be-8d6f-b90f1848d640/dns/0.log"
Nov 01 19:58:53.547215 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:53.547186    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6svs9_6533ed5f-4009-45be-8d6f-b90f1848d640/kube-rbac-proxy/0.log"
Nov 01 19:58:53.634717 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:53.634617    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc47fccc9-z2fz4" event={"ID":"418a4dcb-fb3f-4917-9da9-cf43e683e335","Type":"ContainerStarted","Data":"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"}
Nov 01 19:58:53.653245 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:53.653196    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-dc47fccc9-z2fz4" podStartSLOduration=1.321441247 podStartE2EDuration="4.653181239s" totalImagesPullingTime="3.331739992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:53.652383758 +0000 UTC m=+21.373462434" watchObservedRunningTime="2026-11-01 19:58:53.653181239 +0000 UTC m=+21.374259914"
Nov 01 19:58:54.347325 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:54.347291    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6p64_9666c411-ce1d-4592-82de-51b88b02ea47/dns-node-resolver/0.log"
Nov 01 19:58:54.747102 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:54.747018    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59b96df89b-pmb2t_60703b7e-ae49-4bec-923c-73ade0101e57/registry/0.log"
Nov 01 19:58:54.946560 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:54.946529    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77f5b9cf45-wqbcb_c8da3aea-f54a-45ad-b9bf-4a8108f7951e/registry/0.log"
Nov 01 19:58:55.147193 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:55.147163    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bb76fff67-jsvjg_03bbc2e4-cb60-4916-ad80-b38466c75efd/registry/0.log"
Nov 01 19:58:55.746436 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:55.746406    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szb2b_f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/node-ca/0.log"
Nov 01 19:58:55.946836 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:55.946805    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 19:58:56.147140 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.147103    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74zbl_a5aab77d-3a73-46b8-ba03-03e96ebf16f0/serve-healthcheck-canary/0.log"
Nov 01 19:58:56.439539 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.439467    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fv2q4"]
Nov 01 19:58:56.461228 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.461195    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.509489 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.509441    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 19:58:56.509646 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.509544    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 19:58:56.509646 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.509621    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-88k4l\""
Nov 01 19:58:56.527698 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.527668    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 19:58:56.527968 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.527763    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 19:58:56.653658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.653658 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653663    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-sys\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654068 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653691    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-textfile\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654068 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654068 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653903    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-root\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654068 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.653933    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-wtmp\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654068 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.654065    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgbk\" (UniqueName: \"kubernetes.io/projected/5d3dfa71-36f6-448b-9339-750b874647e0-kube-api-access-lvgbk\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654380 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.654096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-tls\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.654380 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.654155    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-metrics-client-ca\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755677 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755591    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-metrics-client-ca\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755677 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755680    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-sys\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755700    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-textfile\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755741    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-root\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755756    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-wtmp\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755850    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgbk\" (UniqueName: \"kubernetes.io/projected/5d3dfa71-36f6-448b-9339-750b874647e0-kube-api-access-lvgbk\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.755933 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.755878    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-tls\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.756539 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.756397    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-sys\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.756539 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.756472    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-root\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.756672 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.756601    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-textfile\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.756735 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.756714    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-wtmp\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.756919 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.756894    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-metrics-client-ca\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.757368 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.757330    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.759460 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.759431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.759562 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.759493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d3dfa71-36f6-448b-9339-750b874647e0-node-exporter-tls\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.765692 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.765669    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgbk\" (UniqueName: \"kubernetes.io/projected/5d3dfa71-36f6-448b-9339-750b874647e0-kube-api-access-lvgbk\") pod \"node-exporter-fv2q4\" (UID: \"5d3dfa71-36f6-448b-9339-750b874647e0\") " pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.818572 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:56.818541    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fv2q4"
Nov 01 19:58:56.860292 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:58:56.860259    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3dfa71_36f6_448b_9339_750b874647e0.slice/crio-936f75e9831fd4fc22c909582aef6537d41e715294b4e7089bfa56313147e35a WatchSource:0}: Error finding container 936f75e9831fd4fc22c909582aef6537d41e715294b4e7089bfa56313147e35a: Status 404 returned error can't find the container with id 936f75e9831fd4fc22c909582aef6537d41e715294b4e7089bfa56313147e35a
Nov 01 19:58:57.658284 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:57.658241    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv2q4" event={"ID":"5d3dfa71-36f6-448b-9339-750b874647e0","Type":"ContainerStarted","Data":"936f75e9831fd4fc22c909582aef6537d41e715294b4e7089bfa56313147e35a"}
Nov 01 19:58:57.935978 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:57.935896    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-789kh"
Nov 01 19:58:58.550022 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:58:58.549986    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 19:58:58.665737 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:58.665699    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv2q4" event={"ID":"5d3dfa71-36f6-448b-9339-750b874647e0","Type":"ContainerStarted","Data":"c9ccf9345aaff194ba3d0ba7638e3619549580af9b83443fb201fcaa36116a3e"}
Nov 01 19:58:59.423273 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.423242    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:58:59.449124 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.449095    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qdn6f"
Nov 01 19:58:59.672030 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.671998    5041 generic.go:356] "Generic (PLEG): container finished" podID="5d3dfa71-36f6-448b-9339-750b874647e0" containerID="c9ccf9345aaff194ba3d0ba7638e3619549580af9b83443fb201fcaa36116a3e" exitCode=0
Nov 01 19:58:59.672460 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.672082    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv2q4" event={"ID":"5d3dfa71-36f6-448b-9339-750b874647e0","Type":"ContainerDied","Data":"c9ccf9345aaff194ba3d0ba7638e3619549580af9b83443fb201fcaa36116a3e"}
Nov 01 19:58:59.752518 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.752483    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:59.752634 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.752537    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:58:59.758214 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:58:59.758191    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:59:00.679077 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:00.679040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv2q4" event={"ID":"5d3dfa71-36f6-448b-9339-750b874647e0","Type":"ContainerStarted","Data":"5d55901d2de99bd0acad9f3b86b8b0f1735d9be355ad6876fe0c23cdbf4cf46c"}
Nov 01 19:59:00.679561 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:00.679098    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fv2q4" event={"ID":"5d3dfa71-36f6-448b-9339-750b874647e0","Type":"ContainerStarted","Data":"728bb256a7e1d033a27a37142aabd16033b625fb19ab8e813ce45ed040842104"}
Nov 01 19:59:00.683417 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:00.683390    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:59:00.701157 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:00.701105    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fv2q4" podStartSLOduration=3.017890965 podStartE2EDuration="4.701090507s" totalImagesPullingTime="1.587698641s" totalInitContainerRuntime="95.500901ms" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:00.699148734 +0000 UTC m=+28.420227407" watchObservedRunningTime="2026-11-01 19:59:00.701090507 +0000 UTC m=+28.422169182"
Nov 01 19:59:02.094542 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:02.094512    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:59:03.904879 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:03.904846    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:03.943604 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:03.943567    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:03.943797 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:03.943698    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:03.952636 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:03.952590    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 19:59:04.028116 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028080    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mmm\" (UniqueName: \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028116 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028124    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028318 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028152    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028318 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028292    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028397 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028331    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028452 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028436    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028510 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.028566 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.028514    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129696 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129656    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129696 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129754    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129910    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mmm\" (UniqueName: \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129932    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.129969 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.129958    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.130623 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.130590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.130874 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.130847    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.130999 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.130975    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.131555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.131523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.132090 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.132063    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.133037 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.133020    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.133227 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.133208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.142649 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.142627    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mmm\" (UniqueName: \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\") pod \"console-6cb98d4987-9x65r\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") " pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.275937 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.275852    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:04.579715 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.579667    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:04.583083 ip-10-0-137-175 kubenswrapper[5041]: W1101 19:59:04.583056    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff838d8a_28dd_4814_8717_88406425696e.slice/crio-580b69b25855218abf6db8ab53334a657fdeb82cf64f361bed3cf6678de355db WatchSource:0}: Error finding container 580b69b25855218abf6db8ab53334a657fdeb82cf64f361bed3cf6678de355db: Status 404 returned error can't find the container with id 580b69b25855218abf6db8ab53334a657fdeb82cf64f361bed3cf6678de355db
Nov 01 19:59:04.701321 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.701291    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb98d4987-9x65r" event={"ID":"ff838d8a-28dd-4814-8717-88406425696e","Type":"ContainerStarted","Data":"d761a13dcb93beb611137ec6455549ff88fc02835eed79f2114396e4a7e09781"}
Nov 01 19:59:04.701461 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.701350    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb98d4987-9x65r" event={"ID":"ff838d8a-28dd-4814-8717-88406425696e","Type":"ContainerStarted","Data":"580b69b25855218abf6db8ab53334a657fdeb82cf64f361bed3cf6678de355db"}
Nov 01 19:59:04.719895 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:04.719839    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cb98d4987-9x65r" podStartSLOduration=1.719822364 podStartE2EDuration="1.719822364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:04.718629826 +0000 UTC m=+32.439708503" watchObservedRunningTime="2026-11-01 19:59:04.719822364 +0000 UTC m=+32.440901040"
Nov 01 19:59:07.412170 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:07.412135    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5t7b"
Nov 01 19:59:07.718187 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:07.718096    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6bdrz" event={"ID":"f631778e-0855-4a7f-8e2d-6c0385f9bbbe","Type":"ContainerStarted","Data":"b63ee6448ed38eef1704950e7cdcd0e496fab669514444e359faeb8733fa3508"}
Nov 01 19:59:07.718472 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:07.718309    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:59:07.734186 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:07.734160    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6bdrz"
Nov 01 19:59:07.736576 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:07.736531    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6bdrz" podStartSLOduration=1.7416377280000002 podStartE2EDuration="18.736519762s" totalImagesPullingTime="16.994882034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:07.734385406 +0000 UTC m=+35.455464085" watchObservedRunningTime="2026-11-01 19:59:07.736519762 +0000 UTC m=+35.457598437"
Nov 01 19:59:08.682466 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:08.682431    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:11.633560 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:11.633526    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bb76fff67-jsvjg"
Nov 01 19:59:11.683699 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:11.683626    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:59:14.276324 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.276235    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:14.448426 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.448383    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry" containerID="cri-o://f8e6d3e7ea03c51c76fc2b7511908030e4cf8776eb315f2f97917b69817b4df2" gracePeriod=30
Nov 01 19:59:14.762444 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.762405    5041 generic.go:356] "Generic (PLEG): container finished" podID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerID="f8e6d3e7ea03c51c76fc2b7511908030e4cf8776eb315f2f97917b69817b4df2" exitCode=0
Nov 01 19:59:14.762601 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.762476    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" event={"ID":"c8da3aea-f54a-45ad-b9bf-4a8108f7951e","Type":"ContainerDied","Data":"f8e6d3e7ea03c51c76fc2b7511908030e4cf8776eb315f2f97917b69817b4df2"}
Nov 01 19:59:14.762601 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.762527    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb" event={"ID":"c8da3aea-f54a-45ad-b9bf-4a8108f7951e","Type":"ContainerDied","Data":"1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6"}
Nov 01 19:59:14.762601 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.762549    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d04e60403726873ba04f349eb7edaa8fdd5f45b5d5da54e6bc3207b3da2b1c6"
Nov 01 19:59:14.788675 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.788648    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:59:14.843317 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843242    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843317 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843296    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843342    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843408    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843456    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843493    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843555 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843537    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.843839 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.843607    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") pod \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\" (UID: \"c8da3aea-f54a-45ad-b9bf-4a8108f7951e\") "
Nov 01 19:59:14.844239 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.844167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:14.844353 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.844234    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:14.847056 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.847019    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:14.847202 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.847159    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "kube-api-access-ffqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:14.847428 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.847398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:14.854280 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.854250    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:14.856738 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.856714    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:14.858180 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.858158    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls" pod "c8da3aea-f54a-45ad-b9bf-4a8108f7951e" (UID: "c8da3aea-f54a-45ad-b9bf-4a8108f7951e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:14.944988 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.944945    5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-installation-pull-secrets\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.944988 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.944981    5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-certificates\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.944988 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.944993    5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-bound-sa-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.945255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.945003    5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-image-registry-private-configuration\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.945255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.945014    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-trusted-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.945255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.945024    5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-registry-tls\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.945255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.945032    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqsb\" (UniqueName: \"kubernetes.io/projected/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-kube-api-access-ffqsb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:14.945255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:14.945041    5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c8da3aea-f54a-45ad-b9bf-4a8108f7951e-ca-trust-extracted\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:15.528684 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:15.528650    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:59:15.768418 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:15.768391    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77f5b9cf45-wqbcb"
Nov 01 19:59:15.793464 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:15.793437    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:59:15.802973 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:15.802949    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77f5b9cf45-wqbcb"]
Nov 01 19:59:16.426435 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:16.426404    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" path="/var/lib/kubelet/pods/c8da3aea-f54a-45ad-b9bf-4a8108f7951e/volumes"
Nov 01 19:59:20.725940 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:20.725902    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:59:20.726439 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:20.726102    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-t2q78-k5stm" podUID="c769b88a-d848-424a-bacb-79d80ea49e48" containerName="insights-gathering" containerID="cri-o://ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf" gracePeriod=30
Nov 01 19:59:21.019158 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.019134    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:59:21.097602 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097564    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") pod \"c769b88a-d848-424a-bacb-79d80ea49e48\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") "
Nov 01 19:59:21.097749 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097616    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") pod \"c769b88a-d848-424a-bacb-79d80ea49e48\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") "
Nov 01 19:59:21.097749 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097644    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") pod \"c769b88a-d848-424a-bacb-79d80ea49e48\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") "
Nov 01 19:59:21.097749 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097702    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") pod \"c769b88a-d848-424a-bacb-79d80ea49e48\" (UID: \"c769b88a-d848-424a-bacb-79d80ea49e48\") "
Nov 01 19:59:21.098054 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097965    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle" pod "c769b88a-d848-424a-bacb-79d80ea49e48" (UID: "c769b88a-d848-424a-bacb-79d80ea49e48"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:21.098162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.097981    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path" pod "c769b88a-d848-424a-bacb-79d80ea49e48" (UID: "c769b88a-d848-424a-bacb-79d80ea49e48"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:21.098162 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.098113    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle" pod "c769b88a-d848-424a-bacb-79d80ea49e48" (UID: "c769b88a-d848-424a-bacb-79d80ea49e48"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:21.100033 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.100012    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf" pod "c769b88a-d848-424a-bacb-79d80ea49e48" (UID: "c769b88a-d848-424a-bacb-79d80ea49e48"). InnerVolumeSpecName "kube-api-access-4p4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:21.198571 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.198541    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-service-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:21.198571 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.198568    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c769b88a-d848-424a-bacb-79d80ea49e48-archives-path\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:21.198741 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.198578    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p4hf\" (UniqueName: \"kubernetes.io/projected/c769b88a-d848-424a-bacb-79d80ea49e48-kube-api-access-4p4hf\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:21.198741 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.198589    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c769b88a-d848-424a-bacb-79d80ea49e48-trusted-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:21.795222 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.795187    5041 generic.go:356] "Generic (PLEG): container finished" podID="c769b88a-d848-424a-bacb-79d80ea49e48" containerID="ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf" exitCode=2
Nov 01 19:59:21.795760 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.795233    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-t2q78-k5stm"
Nov 01 19:59:21.795760 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.795242    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t2q78-k5stm" event={"ID":"c769b88a-d848-424a-bacb-79d80ea49e48","Type":"ContainerDied","Data":"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"}
Nov 01 19:59:21.795760 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.795292    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-t2q78-k5stm" event={"ID":"c769b88a-d848-424a-bacb-79d80ea49e48","Type":"ContainerDied","Data":"2657699e1be90d0e0fc371f2efedef8d9b2d2be84e47969ea2f18ee760a136bc"}
Nov 01 19:59:21.795760 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.795319    5041 scope.go:122] "RemoveContainer" containerID="ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"
Nov 01 19:59:21.804629 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.804613    5041 scope.go:122] "RemoveContainer" containerID="ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"
Nov 01 19:59:21.804930 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:59:21.804910    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf\": container with ID starting with ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf not found: ID does not exist" containerID="ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"
Nov 01 19:59:21.804989 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.804939    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf"} err="failed to get container status \"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf\": rpc error: code = NotFound desc = could not find container \"ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf\": container with ID starting with ae758f6a3640f1c3db039a84946bfbc9a60f45aa64c50c0335604ac5e13006cf not found: ID does not exist"
Nov 01 19:59:21.815587 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.815566    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:59:21.821190 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:21.821173    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-t2q78-k5stm"]
Nov 01 19:59:22.425402 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:22.425367    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c769b88a-d848-424a-bacb-79d80ea49e48" path="/var/lib/kubelet/pods/c769b88a-d848-424a-bacb-79d80ea49e48/volumes"
Nov 01 19:59:28.540141 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:59:28.540103    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 19:59:33.706980 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:33.706911    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cb98d4987-9x65r" podUID="ff838d8a-28dd-4814-8717-88406425696e" containerName="console" containerID="cri-o://d761a13dcb93beb611137ec6455549ff88fc02835eed79f2114396e4a7e09781" gracePeriod=15
Nov 01 19:59:33.850428 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:33.850400    5041 generic.go:356] "Generic (PLEG): container finished" podID="ff838d8a-28dd-4814-8717-88406425696e" containerID="d761a13dcb93beb611137ec6455549ff88fc02835eed79f2114396e4a7e09781" exitCode=0
Nov 01 19:59:33.850559 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:33.850447    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb98d4987-9x65r" event={"ID":"ff838d8a-28dd-4814-8717-88406425696e","Type":"ContainerDied","Data":"d761a13dcb93beb611137ec6455549ff88fc02835eed79f2114396e4a7e09781"}
Nov 01 19:59:33.993485 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:33.993456    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:34.103407 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103374    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103416    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103459    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103497    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\" (UniqueName: \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103523    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103579 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103565    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103891 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103615    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103891 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103667    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\") pod \"ff838d8a-28dd-4814-8717-88406425696e\" (UID: \"ff838d8a-28dd-4814-8717-88406425696e\") "
Nov 01 19:59:34.103891 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103773    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:34.104034 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103902    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:34.104034 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103945    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:34.104034 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.103953    5041 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff838d8a-28dd-4814-8717-88406425696e-tmp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.104264 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.104233    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:34.104340 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.104284    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:34.105880 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.105855    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:34.105974 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.105882    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:34.105974 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.105958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm" pod "ff838d8a-28dd-4814-8717-88406425696e" (UID: "ff838d8a-28dd-4814-8717-88406425696e"). InnerVolumeSpecName "kube-api-access-b8mmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:34.204628 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204597    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-trusted-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204628 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204624    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mmm\" (UniqueName: \"kubernetes.io/projected/ff838d8a-28dd-4814-8717-88406425696e-kube-api-access-b8mmm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204628 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204636    5041 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-serving-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204868 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204645    5041 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff838d8a-28dd-4814-8717-88406425696e-console-oauth-config\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204868 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204655    5041 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-oauth-serving-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204868 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204664    5041 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-console-config\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.204868 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.204672    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff838d8a-28dd-4814-8717-88406425696e-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:34.856284 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.856253    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cb98d4987-9x65r"
Nov 01 19:59:34.856682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.856284    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cb98d4987-9x65r" event={"ID":"ff838d8a-28dd-4814-8717-88406425696e","Type":"ContainerDied","Data":"580b69b25855218abf6db8ab53334a657fdeb82cf64f361bed3cf6678de355db"}
Nov 01 19:59:34.856682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.856339    5041 scope.go:122] "RemoveContainer" containerID="d761a13dcb93beb611137ec6455549ff88fc02835eed79f2114396e4a7e09781"
Nov 01 19:59:34.872033 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.872012    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:34.877347 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:34.877325    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cb98d4987-9x65r"]
Nov 01 19:59:36.428511 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:36.428476    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff838d8a-28dd-4814-8717-88406425696e" path="/var/lib/kubelet/pods/ff838d8a-28dd-4814-8717-88406425696e/volumes"
Nov 01 19:59:36.711239 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:36.711135    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry" containerID="cri-o://f5e1113216c2068efe3417ff2d4d7d8e5c893bc8d56381d80c18c7dedc7dcd1d" gracePeriod=30
Nov 01 19:59:36.868671 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:36.868635    5041 generic.go:356] "Generic (PLEG): container finished" podID="60703b7e-ae49-4bec-923c-73ade0101e57" containerID="f5e1113216c2068efe3417ff2d4d7d8e5c893bc8d56381d80c18c7dedc7dcd1d" exitCode=0
Nov 01 19:59:36.868876 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:36.868687    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" event={"ID":"60703b7e-ae49-4bec-923c-73ade0101e57","Type":"ContainerDied","Data":"f5e1113216c2068efe3417ff2d4d7d8e5c893bc8d56381d80c18c7dedc7dcd1d"}
Nov 01 19:59:36.990816 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:36.990739    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:59:37.132504 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132465    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132504 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132505    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132717 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132557    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132717 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132601    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132855 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132757    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132855 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132830    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132954 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132877    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.132954 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.132935    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") pod \"60703b7e-ae49-4bec-923c-73ade0101e57\" (UID: \"60703b7e-ae49-4bec-923c-73ade0101e57\") "
Nov 01 19:59:37.133557 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.133481    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:37.133666 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.133557    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:37.135486 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.135457    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:37.135599 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.135496    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:37.135599 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.135519    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:37.135599 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.135533    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:37.135722 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.135600    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "kube-api-access-dg49c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:37.139994 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.139973    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted" pod "60703b7e-ae49-4bec-923c-73ade0101e57" (UID: "60703b7e-ae49-4bec-923c-73ade0101e57"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:37.234140 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234103    5041 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-registry-certificates\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234140 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234134    5041 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/60703b7e-ae49-4bec-923c-73ade0101e57-ca-trust-extracted\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234140 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234148    5041 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-registry-tls\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234157    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg49c\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-kube-api-access-dg49c\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234166    5041 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60703b7e-ae49-4bec-923c-73ade0101e57-bound-sa-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234174    5041 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-installation-pull-secrets\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234183    5041 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/60703b7e-ae49-4bec-923c-73ade0101e57-image-registry-private-configuration\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.234363 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.234192    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60703b7e-ae49-4bec-923c-73ade0101e57-trusted-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:37.874028 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.873984    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t" event={"ID":"60703b7e-ae49-4bec-923c-73ade0101e57","Type":"ContainerDied","Data":"2ab65a94343bd84b16add892f51acefb9d547abcd1913582ddfc7c02667c340c"}
Nov 01 19:59:37.874028 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.874022    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b96df89b-pmb2t"
Nov 01 19:59:37.874482 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.874055    5041 scope.go:122] "RemoveContainer" containerID="f5e1113216c2068efe3417ff2d4d7d8e5c893bc8d56381d80c18c7dedc7dcd1d"
Nov 01 19:59:37.895837 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.895811    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:59:37.899565 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:37.899543    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59b96df89b-pmb2t"]
Nov 01 19:59:38.425660 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:38.425622    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" path="/var/lib/kubelet/pods/60703b7e-ae49-4bec-923c-73ade0101e57/volumes"
Nov 01 19:59:40.549114 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.549077    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-dc47fccc9-z2fz4" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerName="console" containerID="cri-o://2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1" gracePeriod=15
Nov 01 19:59:40.680046 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.680013    5041 patch_prober.go:29] interesting pod/console-dc47fccc9-z2fz4 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.31:8443/health\": dial tcp 10.132.0.31:8443: connect: connection refused" start-of-body=
Nov 01 19:59:40.680196 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.680061    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-dc47fccc9-z2fz4" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerName="console" probeResult="failure" output="Get \"https://10.132.0.31:8443/health\": dial tcp 10.132.0.31:8443: connect: connection refused"
Nov 01 19:59:40.837701 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.837674    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:59:40.888890 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.888854    5041 generic.go:356] "Generic (PLEG): container finished" podID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerID="2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1" exitCode=0
Nov 01 19:59:40.889054 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.888900    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc47fccc9-z2fz4"
Nov 01 19:59:40.889054 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.888942    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc47fccc9-z2fz4" event={"ID":"418a4dcb-fb3f-4917-9da9-cf43e683e335","Type":"ContainerDied","Data":"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"}
Nov 01 19:59:40.889054 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.888994    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc47fccc9-z2fz4" event={"ID":"418a4dcb-fb3f-4917-9da9-cf43e683e335","Type":"ContainerDied","Data":"a209ffce8a488691f5d9c55de4adc4abc95fa6f87bf9c6d1a9b0eaa165d91a26"}
Nov 01 19:59:40.889054 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.889015    5041 scope.go:122] "RemoveContainer" containerID="2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"
Nov 01 19:59:40.896907 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.896882    5041 scope.go:122] "RemoveContainer" containerID="2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"
Nov 01 19:59:40.897187 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:59:40.897168    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1\": container with ID starting with 2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1 not found: ID does not exist" containerID="2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"
Nov 01 19:59:40.897240 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.897195    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1"} err="failed to get container status \"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1\": rpc error: code = NotFound desc = could not find container \"2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1\": container with ID starting with 2453fa5855617e8546850c44d148bb6a1e5d074e6a2bcee05b0e4cd07507dee1 not found: ID does not exist"
Nov 01 19:59:40.966513 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966528    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966559    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\" (UniqueName: \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966631    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966934 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966693    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\" (UniqueName: \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.966934 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966739    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\") pod \"418a4dcb-fb3f-4917-9da9-cf43e683e335\" (UID: \"418a4dcb-fb3f-4917-9da9-cf43e683e335\") "
Nov 01 19:59:40.967028 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.966954    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:40.967080 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.967034    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:40.967080 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.967048    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:40.967080 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.967054    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:40.967239 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.967088    5041 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/418a4dcb-fb3f-4917-9da9-cf43e683e335-tmp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:40.968968 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.968943    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:40.968968 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.968958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 19:59:40.969110 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:40.968985    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8" pod "418a4dcb-fb3f-4917-9da9-cf43e683e335" (UID: "418a4dcb-fb3f-4917-9da9-cf43e683e335"). InnerVolumeSpecName "kube-api-access-6zzx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067602    5041 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-serving-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067627    5041 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-oauth-serving-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067638    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067647    5041 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-config\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067656    5041 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/418a4dcb-fb3f-4917-9da9-cf43e683e335-console-oauth-config\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.067682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.067667    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzx8\" (UniqueName: \"kubernetes.io/projected/418a4dcb-fb3f-4917-9da9-cf43e683e335-kube-api-access-6zzx8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:41.208964 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.208935    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:59:41.212905 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:41.212884    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc47fccc9-z2fz4"]
Nov 01 19:59:42.425595 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:42.425535    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" path="/var/lib/kubelet/pods/418a4dcb-fb3f-4917-9da9-cf43e683e335/volumes"
Nov 01 19:59:42.905523 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:42.905490    5041 generic.go:356] "Generic (PLEG): container finished" podID="adfbd1ed-3a02-4334-b370-328f2a511729" containerID="a52765ad9f86762dd01a5039ff73b6a8f690c2c6a14b5f52dbc3e0ce58c0133f" exitCode=0
Nov 01 19:59:42.905710 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:42.905529    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerDied","Data":"a52765ad9f86762dd01a5039ff73b6a8f690c2c6a14b5f52dbc3e0ce58c0133f"}
Nov 01 19:59:42.905710 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:42.905575    5041 scope.go:122] "RemoveContainer" containerID="c0cfd4fd8aced840fbb1071afcf11574419b4effb12a00fccb6737bd7e18953b"
Nov 01 19:59:42.905953 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:42.905923    5041 scope.go:122] "RemoveContainer" containerID="a52765ad9f86762dd01a5039ff73b6a8f690c2c6a14b5f52dbc3e0ce58c0133f"
Nov 01 19:59:42.906180 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:59:42.906154    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-4rbdb_openshift-insights(adfbd1ed-3a02-4334-b370-328f2a511729)\"" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" podUID="adfbd1ed-3a02-4334-b370-328f2a511729"
Nov 01 19:59:53.421102 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.421070    5041 scope.go:122] "RemoveContainer" containerID="a52765ad9f86762dd01a5039ff73b6a8f690c2c6a14b5f52dbc3e0ce58c0133f"
Nov 01 19:59:53.797665 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.797622    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:59:53.797894 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.797869    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-hclz8-58g2v" podUID="552dbe68-8fe8-4a86-b74a-49d441b5075a" containerName="insights-gathering" containerID="cri-o://daf4cc2c598e45427e22489451814d67b1b4e68954f7b9eb69ff6bd3f454f7c3" gracePeriod=30
Nov 01 19:59:53.953682 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.953645    5041 generic.go:356] "Generic (PLEG): container finished" podID="552dbe68-8fe8-4a86-b74a-49d441b5075a" containerID="daf4cc2c598e45427e22489451814d67b1b4e68954f7b9eb69ff6bd3f454f7c3" exitCode=2
Nov 01 19:59:53.953871 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.953717    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hclz8-58g2v" event={"ID":"552dbe68-8fe8-4a86-b74a-49d441b5075a","Type":"ContainerDied","Data":"daf4cc2c598e45427e22489451814d67b1b4e68954f7b9eb69ff6bd3f454f7c3"}
Nov 01 19:59:53.955811 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:53.955762    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4rbdb" event={"ID":"adfbd1ed-3a02-4334-b370-328f2a511729","Type":"ContainerStarted","Data":"723510f42da1a63c8c11c3d7369f789163b51d216076ddf83efccceed19634d0"}
Nov 01 19:59:54.040976 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.040945    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:59:54.070255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.070176    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\" (UniqueName: \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\") pod \"552dbe68-8fe8-4a86-b74a-49d441b5075a\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") "
Nov 01 19:59:54.070255 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.070209    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\") pod \"552dbe68-8fe8-4a86-b74a-49d441b5075a\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") "
Nov 01 19:59:54.070592 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.070561    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle" pod "552dbe68-8fe8-4a86-b74a-49d441b5075a" (UID: "552dbe68-8fe8-4a86-b74a-49d441b5075a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:54.072232 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.072206    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt" pod "552dbe68-8fe8-4a86-b74a-49d441b5075a" (UID: "552dbe68-8fe8-4a86-b74a-49d441b5075a"). InnerVolumeSpecName "kube-api-access-lcwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 19:59:54.171119 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171082    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\") pod \"552dbe68-8fe8-4a86-b74a-49d441b5075a\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") "
Nov 01 19:59:54.171119 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171121    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\") pod \"552dbe68-8fe8-4a86-b74a-49d441b5075a\" (UID: \"552dbe68-8fe8-4a86-b74a-49d441b5075a\") "
Nov 01 19:59:54.171324 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171243    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcwpt\" (UniqueName: \"kubernetes.io/projected/552dbe68-8fe8-4a86-b74a-49d441b5075a-kube-api-access-lcwpt\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:54.171324 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171257    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-service-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:54.171459 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171435    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle" pod "552dbe68-8fe8-4a86-b74a-49d441b5075a" (UID: "552dbe68-8fe8-4a86-b74a-49d441b5075a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 19:59:54.171491 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.171445    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path" pod "552dbe68-8fe8-4a86-b74a-49d441b5075a" (UID: "552dbe68-8fe8-4a86-b74a-49d441b5075a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 19:59:54.271573 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.271539    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552dbe68-8fe8-4a86-b74a-49d441b5075a-trusted-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:54.271573 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.271570    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552dbe68-8fe8-4a86-b74a-49d441b5075a-archives-path\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 19:59:54.961404 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.961364    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-hclz8-58g2v" event={"ID":"552dbe68-8fe8-4a86-b74a-49d441b5075a","Type":"ContainerDied","Data":"3b305d68c47ad2f4288fd4b61c3edc98176216901b1e5782ce1fe1f032f4844e"}
Nov 01 19:59:54.961404 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.961390    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-hclz8-58g2v"
Nov 01 19:59:54.961945 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.961431    5041 scope.go:122] "RemoveContainer" containerID="daf4cc2c598e45427e22489451814d67b1b4e68954f7b9eb69ff6bd3f454f7c3"
Nov 01 19:59:54.976153 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.976129    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:59:54.978423 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:54.978401    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-hclz8-58g2v"]
Nov 01 19:59:56.426042 ip-10-0-137-175 kubenswrapper[5041]: I1101 19:59:56.426010    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552dbe68-8fe8-4a86-b74a-49d441b5075a" path="/var/lib/kubelet/pods/552dbe68-8fe8-4a86-b74a-49d441b5075a/volumes"
Nov 01 19:59:58.539985 ip-10-0-137-175 kubenswrapper[5041]: E1101 19:59:58.539951    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:00:28.539812 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:00:28.539753    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:00:58.539236 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:00:58.539163    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:01:07.395756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.395717    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"]
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396248    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396270    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396291    5041 cpu_manager.go:414] "containerMap: removing container" podUID="552dbe68-8fe8-4a86-b74a-49d441b5075a" containerName="insights-gathering"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396300    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552dbe68-8fe8-4a86-b74a-49d441b5075a" containerName="insights-gathering"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396315    5041 cpu_manager.go:414] "containerMap: removing container" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerName="console"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396324    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerName="console"
Nov 01 20:01:07.396350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396348    5041 cpu_manager.go:414] "containerMap: removing container" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396356    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396370    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ff838d8a-28dd-4814-8717-88406425696e" containerName="console"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396378    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff838d8a-28dd-4814-8717-88406425696e" containerName="console"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396394    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c769b88a-d848-424a-bacb-79d80ea49e48" containerName="insights-gathering"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396401    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c769b88a-d848-424a-bacb-79d80ea49e48" containerName="insights-gathering"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396489    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="60703b7e-ae49-4bec-923c-73ade0101e57" containerName="registry"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396502    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff838d8a-28dd-4814-8717-88406425696e" containerName="console"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396511    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c769b88a-d848-424a-bacb-79d80ea49e48" containerName="insights-gathering"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396524    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8da3aea-f54a-45ad-b9bf-4a8108f7951e" containerName="registry"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396533    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="552dbe68-8fe8-4a86-b74a-49d441b5075a" containerName="insights-gathering"
Nov 01 20:01:07.396603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.396543    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="418a4dcb-fb3f-4917-9da9-cf43e683e335" containerName="console"
Nov 01 20:01:07.399291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.399269    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.413109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.413085    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"]
Nov 01 20:01:07.421159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.421140    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-q9rl7\""
Nov 01 20:01:07.421367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.421354    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 20:01:07.476742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.476704    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"]
Nov 01 20:01:07.479140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.479118    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.487283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.487249    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"]
Nov 01 20:01:07.520979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.520953    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-55bvj\""
Nov 01 20:01:07.521455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.521437    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 20:01:07.521523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.521475    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 20:01:07.566790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.566755    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.566886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.566813    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5848925e-fc2c-43d4-b75f-a3b29a194244-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.566930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.566894    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.566930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.566917    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.566996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.566940    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.567040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.567006    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.567110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.567096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2r9s\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-kube-api-access-h2r9s\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.567153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.567143    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.567187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.567165    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.667961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.667889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.667961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.667918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.667961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.667937    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668049    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668084    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.668157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668129    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2r9s\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-kube-api-access-h2r9s\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668236    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.668436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668335    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvw6\" (UniqueName: \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.668436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668390    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668412    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668474    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668512    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5848925e-fc2c-43d4-b75f-a3b29a194244-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668706    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668827    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.668922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.668907    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.669143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.669126    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5848925e-fc2c-43d4-b75f-a3b29a194244-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.670235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.670214    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.670582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.670563    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.675736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.675709    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.675872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.675854    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2r9s\" (UniqueName: \"kubernetes.io/projected/5848925e-fc2c-43d4-b75f-a3b29a194244-kube-api-access-h2r9s\") pod \"openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv\" (UID: \"5848925e-fc2c-43d4-b75f-a3b29a194244\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.712605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.712581    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:07.769686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.769647    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.769832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.769725    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.769832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.769756    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvw6\" (UniqueName: \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.770616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.770579    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.770926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.770903    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.777946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.777926    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvw6\" (UniqueName: \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.792744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.792722    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:07.984755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:07.984733    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"]
Nov 01 20:01:07.986906 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:07.986878    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5848925e_fc2c_43d4_b75f_a3b29a194244.slice/crio-8b469e7b1de4233117601d0cc86fe2f3f89e7f081ea4a83d00ff29be142b07bd WatchSource:0}: Error finding container 8b469e7b1de4233117601d0cc86fe2f3f89e7f081ea4a83d00ff29be142b07bd: Status 404 returned error can't find the container with id 8b469e7b1de4233117601d0cc86fe2f3f89e7f081ea4a83d00ff29be142b07bd
Nov 01 20:01:08.059412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:08.059377    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"]
Nov 01 20:01:08.062369 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:08.062341    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819b6f5d_97fb_4bd6_a3c1_1227ca0456f0.slice/crio-dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6 WatchSource:0}: Error finding container dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6: Status 404 returned error can't find the container with id dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6
Nov 01 20:01:08.250831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:08.250732    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv" event={"ID":"5848925e-fc2c-43d4-b75f-a3b29a194244","Type":"ContainerStarted","Data":"8b469e7b1de4233117601d0cc86fe2f3f89e7f081ea4a83d00ff29be142b07bd"}
Nov 01 20:01:08.251889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:08.251852    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn" event={"ID":"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0","Type":"ContainerStarted","Data":"dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6"}
Nov 01 20:01:09.813634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.812200    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"]
Nov 01 20:01:09.815655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.815633    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.819999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.819974    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xvn5p\""
Nov 01 20:01:09.824312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.824285    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"]
Nov 01 20:01:09.836293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.834821    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"]
Nov 01 20:01:09.837328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.837310    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.845971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.845825    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"]
Nov 01 20:01:09.869176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.869145    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"]
Nov 01 20:01:09.872610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.872094    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.885349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885321    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.885479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885366    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.885534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885476    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.885534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885522    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxdzv\" (UniqueName: \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.885608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885561    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.885608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885597    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.885684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885632    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.885684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885678    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8nt\" (UniqueName: \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.885756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885703    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqgz\" (UniqueName: \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.885995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.885970    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"]
Nov 01 20:01:09.919707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.919675    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vzccx\""
Nov 01 20:01:09.919863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.919738    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4nxrp\""
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.986923    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.986979    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxdzv\" (UniqueName: \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987099    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987143    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8nt\" (UniqueName: \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987216    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqgz\" (UniqueName: \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987288    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987319    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987375    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.987757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987583    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.988393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987859    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:09.988393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.988092    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.988393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.988104    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.988607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.987679    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.996501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.996432    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxdzv\" (UniqueName: \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:09.996501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.996435    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8nt\" (UniqueName: \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:09.996949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:09.996921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqgz\" (UniqueName: \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:10.134825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.134694    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:10.157708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.157676    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:10.190031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.189993    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:10.420032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.420006    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"]
Nov 01 20:01:10.462399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.462319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"]
Nov 01 20:01:10.465549 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:10.465517    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551eb86c_2762_48d8_8dad_63df589a1c4d.slice/crio-e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d WatchSource:0}: Error finding container e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d: Status 404 returned error can't find the container with id e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d
Nov 01 20:01:10.480079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:10.480055    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"]
Nov 01 20:01:10.483666 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:10.483644    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb9b63e_0bd7_4db3_83eb_64f83d6fab5c.slice/crio-7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba WatchSource:0}: Error finding container 7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba: Status 404 returned error can't find the container with id 7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba
Nov 01 20:01:11.274414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:11.274373    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerStarted","Data":"7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba"}
Nov 01 20:01:11.276066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:11.276035    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerStarted","Data":"e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d"}
Nov 01 20:01:11.278108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:11.278080    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerStarted","Data":"5ea855c3b41055e4459ee281e2e9718bd6dcf8ff6fda8ddbc9d114a8ab22341a"}
Nov 01 20:01:12.063129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.063076    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:01:12.063262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.063171    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:01:12.063262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.063213    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:01:12.290806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.289419    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv" event={"ID":"5848925e-fc2c-43d4-b75f-a3b29a194244","Type":"ContainerStarted","Data":"8c97885fc52208b214ed24f0889505e4bad626f1ad3c09c81f6f29a6bff6a4b4"}
Nov 01 20:01:12.313724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.312076    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv" podStartSLOduration=1.238962023 podStartE2EDuration="5.312056004s" totalImagesPullingTime="4.073093981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:12.310592353 +0000 UTC m=+160.031671031" watchObservedRunningTime="2026-11-01 20:01:12.312056004 +0000 UTC m=+160.033134681"
Nov 01 20:01:12.713316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:12.713232    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:13.718620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:13.718577    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:14.300594 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:14.300490    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:14.302261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:14.301946    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv"
Nov 01 20:01:17.323277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:17.321500    5041 generic.go:356] "Generic (PLEG): container finished" podID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerID="433c522901afc53550e259b2f2f185e35708f9d0b09834fab3392e26d7d78209" exitCode=0
Nov 01 20:01:17.323277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:17.321569    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn" event={"ID":"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0","Type":"ContainerDied","Data":"433c522901afc53550e259b2f2f185e35708f9d0b09834fab3392e26d7d78209"}
Nov 01 20:01:18.329328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:18.329289    5041 generic.go:356] "Generic (PLEG): container finished" podID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerID="8bb12a3ef55e1188d909c6767daf60dc6533083ec53be63a72433e0cbb0c37bb" exitCode=0
Nov 01 20:01:18.329826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:18.329383    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn" event={"ID":"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0","Type":"ContainerDied","Data":"8bb12a3ef55e1188d909c6767daf60dc6533083ec53be63a72433e0cbb0c37bb"}
Nov 01 20:01:19.340238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:19.340043    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerStarted","Data":"5d863082063212d39a89c5be1d396cdfb816b3ffc57837bfc6bba2571b1a72dd"}
Nov 01 20:01:19.343970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:19.343162    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerStarted","Data":"ef14a1eeb5a61d24b48fa22581adfbc012a7176d684b9ee2437b1f680652f4c7"}
Nov 01 20:01:19.344758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:19.344731    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerStarted","Data":"c7a8357a980e0b0079ac30b23eb815114f0de8b39e4f93b91a603762dea0d02c"}
Nov 01 20:01:20.353089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.353052    5041 generic.go:356] "Generic (PLEG): container finished" podID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerID="5d863082063212d39a89c5be1d396cdfb816b3ffc57837bfc6bba2571b1a72dd" exitCode=0
Nov 01 20:01:20.353719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.353130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerDied","Data":"5d863082063212d39a89c5be1d396cdfb816b3ffc57837bfc6bba2571b1a72dd"}
Nov 01 20:01:20.354883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.354844    5041 generic.go:356] "Generic (PLEG): container finished" podID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerID="ef14a1eeb5a61d24b48fa22581adfbc012a7176d684b9ee2437b1f680652f4c7" exitCode=0
Nov 01 20:01:20.354981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.354925    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerDied","Data":"ef14a1eeb5a61d24b48fa22581adfbc012a7176d684b9ee2437b1f680652f4c7"}
Nov 01 20:01:20.357165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.357142    5041 generic.go:356] "Generic (PLEG): container finished" podID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerID="c7a8357a980e0b0079ac30b23eb815114f0de8b39e4f93b91a603762dea0d02c" exitCode=0
Nov 01 20:01:20.357304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:20.357178    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerDied","Data":"c7a8357a980e0b0079ac30b23eb815114f0de8b39e4f93b91a603762dea0d02c"}
Nov 01 20:01:21.364599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.364510    5041 generic.go:356] "Generic (PLEG): container finished" podID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerID="930412c24d80da65022c1864fd26d8cd38b8093f9c03bbafc2120189c1b40bf5" exitCode=0
Nov 01 20:01:21.364599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.364552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerDied","Data":"930412c24d80da65022c1864fd26d8cd38b8093f9c03bbafc2120189c1b40bf5"}
Nov 01 20:01:21.366936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.366914    5041 generic.go:356] "Generic (PLEG): container finished" podID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerID="98bd25b8856d0765409bb88c1dedcc9ba4e4358db86765d4db825de2b169c374" exitCode=0
Nov 01 20:01:21.367044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.366988    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerDied","Data":"98bd25b8856d0765409bb88c1dedcc9ba4e4358db86765d4db825de2b169c374"}
Nov 01 20:01:21.369227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.369203    5041 generic.go:356] "Generic (PLEG): container finished" podID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerID="e72d6bb79d6ac1ae206538dbe0d767740d10c12931aed2caa90ce52b570733b7" exitCode=0
Nov 01 20:01:21.369332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:21.369234    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerDied","Data":"e72d6bb79d6ac1ae206538dbe0d767740d10c12931aed2caa90ce52b570733b7"}
Nov 01 20:01:27.409175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:27.409141    5041 generic.go:356] "Generic (PLEG): container finished" podID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerID="96486c8a550c7170410a821648b09446a7bdb5803c75c3b2790d012927dea7b0" exitCode=0
Nov 01 20:01:27.409634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:27.409219    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerDied","Data":"96486c8a550c7170410a821648b09446a7bdb5803c75c3b2790d012927dea7b0"}
Nov 01 20:01:27.411369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:27.411302    5041 generic.go:356] "Generic (PLEG): container finished" podID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerID="aec266f36e34e530b3a9c7a97c17ee8ee6e853f062b2574cedc021457bc642a2" exitCode=0
Nov 01 20:01:27.411369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:27.411351    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn" event={"ID":"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0","Type":"ContainerDied","Data":"aec266f36e34e530b3a9c7a97c17ee8ee6e853f062b2574cedc021457bc642a2"}
Nov 01 20:01:28.417856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.417825    5041 generic.go:356] "Generic (PLEG): container finished" podID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerID="56f36df190948363460f9a74e57d2183096c740c672c43b1115ffae870b00b13" exitCode=0
Nov 01 20:01:28.418304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.417915    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerDied","Data":"56f36df190948363460f9a74e57d2183096c740c672c43b1115ffae870b00b13"}
Nov 01 20:01:28.420122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.420096    5041 generic.go:356] "Generic (PLEG): container finished" podID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerID="ed10046512c20eb9b320b3486615e02142aa90349d24b30969a9747d2fc6fcfc" exitCode=0
Nov 01 20:01:28.425415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.425390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerDied","Data":"ed10046512c20eb9b320b3486615e02142aa90349d24b30969a9747d2fc6fcfc"}
Nov 01 20:01:28.552508 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:01:28.552473    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:01:28.569123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.569092    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:28.583510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.583487    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:28.665239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665196    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\" (UniqueName: \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\") pod \"087e3a8c-ee03-4344-a21b-ab602089cc81\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") "
Nov 01 20:01:28.665239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665244    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\") pod \"087e3a8c-ee03-4344-a21b-ab602089cc81\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") "
Nov 01 20:01:28.665530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665268    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\") pod \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") "
Nov 01 20:01:28.665530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\") pod \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") "
Nov 01 20:01:28.665530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665360    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\" (UniqueName: \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\") pod \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\" (UID: \"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0\") "
Nov 01 20:01:28.665530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.665415    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\") pod \"087e3a8c-ee03-4344-a21b-ab602089cc81\" (UID: \"087e3a8c-ee03-4344-a21b-ab602089cc81\") "
Nov 01 20:01:28.666039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.666006    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle" pod "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" (UID: "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:28.666039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.666021    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle" pod "087e3a8c-ee03-4344-a21b-ab602089cc81" (UID: "087e3a8c-ee03-4344-a21b-ab602089cc81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:28.667599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.667570    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv" pod "087e3a8c-ee03-4344-a21b-ab602089cc81" (UID: "087e3a8c-ee03-4344-a21b-ab602089cc81"). InnerVolumeSpecName "kube-api-access-bxdzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:01:28.667927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.667881    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6" pod "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" (UID: "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0"). InnerVolumeSpecName "kube-api-access-qbvw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:01:28.670329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.670301    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util" pod "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" (UID: "819b6f5d-97fb-4bd6-a3c1-1227ca0456f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:28.670712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.670688    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util" pod "087e3a8c-ee03-4344-a21b-ab602089cc81" (UID: "087e3a8c-ee03-4344-a21b-ab602089cc81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:28.766065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766021    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-util\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:28.766065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766059    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxdzv\" (UniqueName: \"kubernetes.io/projected/087e3a8c-ee03-4344-a21b-ab602089cc81-kube-api-access-bxdzv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:28.766065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766069    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/087e3a8c-ee03-4344-a21b-ab602089cc81-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:28.766300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766078    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:28.766300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766087    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-util\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:28.766300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:28.766094    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvw6\" (UniqueName: \"kubernetes.io/projected/819b6f5d-97fb-4bd6-a3c1-1227ca0456f0-kube-api-access-qbvw6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.427725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.427687    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d"
Nov 01 20:01:29.427725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.427710    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjrc2d" event={"ID":"087e3a8c-ee03-4344-a21b-ab602089cc81","Type":"ContainerDied","Data":"5ea855c3b41055e4459ee281e2e9718bd6dcf8ff6fda8ddbc9d114a8ab22341a"}
Nov 01 20:01:29.428237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.427757    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea855c3b41055e4459ee281e2e9718bd6dcf8ff6fda8ddbc9d114a8ab22341a"
Nov 01 20:01:29.429744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.429719    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn"
Nov 01 20:01:29.429871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.429745    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdlmn" event={"ID":"819b6f5d-97fb-4bd6-a3c1-1227ca0456f0","Type":"ContainerDied","Data":"dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6"}
Nov 01 20:01:29.429871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.429810    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd5f81cd5f33a7c38de41d0c335b9e179cf9cedecae6dc74f28bdf35d9c737c6"
Nov 01 20:01:29.608173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.608140    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:29.624049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.624017    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:29.672656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672627    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\" (UniqueName: \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\") pod \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") "
Nov 01 20:01:29.672852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672672    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\") pod \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") "
Nov 01 20:01:29.672852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672705    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\") pod \"551eb86c-2762-48d8-8dad-63df589a1c4d\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") "
Nov 01 20:01:29.672852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672723    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\") pod \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\" (UID: \"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c\") "
Nov 01 20:01:29.672852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672768    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\" (UniqueName: \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\") pod \"551eb86c-2762-48d8-8dad-63df589a1c4d\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") "
Nov 01 20:01:29.672852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.672821    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\") pod \"551eb86c-2762-48d8-8dad-63df589a1c4d\" (UID: \"551eb86c-2762-48d8-8dad-63df589a1c4d\") "
Nov 01 20:01:29.673319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.673290    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle" pod "551eb86c-2762-48d8-8dad-63df589a1c4d" (UID: "551eb86c-2762-48d8-8dad-63df589a1c4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:29.673678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.673644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle" pod "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" (UID: "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:29.674987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.674960    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz" pod "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" (UID: "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c"). InnerVolumeSpecName "kube-api-access-jdqgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:01:29.675401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.675377    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt" pod "551eb86c-2762-48d8-8dad-63df589a1c4d" (UID: "551eb86c-2762-48d8-8dad-63df589a1c4d"). InnerVolumeSpecName "kube-api-access-qf8nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:01:29.677623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.677594    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util" pod "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" (UID: "4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:29.677957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.677909    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util" pod "551eb86c-2762-48d8-8dad-63df589a1c4d" (UID: "551eb86c-2762-48d8-8dad-63df589a1c4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:01:29.773817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773756    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf8nt\" (UniqueName: \"kubernetes.io/projected/551eb86c-2762-48d8-8dad-63df589a1c4d-kube-api-access-qf8nt\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.773817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773812    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-util\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.773817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773822    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdqgz\" (UniqueName: \"kubernetes.io/projected/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-kube-api-access-jdqgz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.774056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773831    5041 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-util\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.774056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773839    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/551eb86c-2762-48d8-8dad-63df589a1c4d-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:29.774056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:29.773848    5041 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:01:30.436496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.436473    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv"
Nov 01 20:01:30.436918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.436489    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajnkjv" event={"ID":"4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c","Type":"ContainerDied","Data":"7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba"}
Nov 01 20:01:30.436918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.436547    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eb5743b1718a573b68f94b70b0412e0cc74a96e1331d13821b2f9767d2f2dba"
Nov 01 20:01:30.438576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.438554    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79" event={"ID":"551eb86c-2762-48d8-8dad-63df589a1c4d","Type":"ContainerDied","Data":"e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d"}
Nov 01 20:01:30.438576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.438571    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6spj79"
Nov 01 20:01:30.438576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:30.438588    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e303f590adb3b9c9201d45b50cff54fe466e550771b0eb0a6c491a3d005ba31d"
Nov 01 20:01:34.627377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627340    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"]
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627755    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="pull"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627769    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="pull"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627800    5041 cpu_manager.go:414] "containerMap: removing container" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627807    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627823    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627828    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627835    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="util"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627840    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="util"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627850    5041 cpu_manager.go:414] "containerMap: removing container" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627855    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="extract"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627863    5041 cpu_manager.go:414] "containerMap: removing container" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="util"
Nov 01 20:01:34.627861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627868    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="util"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627882    5041 cpu_manager.go:414] "containerMap: removing container" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627887    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627895    5041 cpu_manager.go:414] "containerMap: removing container" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="extract"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627899    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="extract"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627907    5041 cpu_manager.go:414] "containerMap: removing container" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627911    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627918    5041 cpu_manager.go:414] "containerMap: removing container" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="util"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627922    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="util"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627928    5041 cpu_manager.go:414] "containerMap: removing container" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627932    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="pull"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627938    5041 cpu_manager.go:414] "containerMap: removing container" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="util"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627942    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="util"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.627991    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="819b6f5d-97fb-4bd6-a3c1-1227ca0456f0" containerName="extract"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.628000    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="551eb86c-2762-48d8-8dad-63df589a1c4d" containerName="extract"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.628007    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb9b63e-0bd7-4db3-83eb-64f83d6fab5c" containerName="extract"
Nov 01 20:01:34.628216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.628012    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="087e3a8c-ee03-4344-a21b-ab602089cc81" containerName="extract"
Nov 01 20:01:34.630725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.630708    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.639571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.639548    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 20:01:34.651891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.651859    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rw9ss\""
Nov 01 20:01:34.651891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.651861    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 20:01:34.651891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.651880    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 20:01:34.652117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.651894    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 20:01:34.670902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.670872    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"]
Nov 01 20:01:34.714985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.714951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.715129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.714995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.715129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.715037    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.715129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.715104    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/2294303a-f8f8-439d-869a-a16d8fe60a1a-kube-api-access-nknzg\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.715236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.715132    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.715236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.715151    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.739668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.739644    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 20:01:34.816061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816032    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816070    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/2294303a-f8f8-439d-869a-a16d8fe60a1a-kube-api-access-nknzg\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816106    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816156    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816184    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.816548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.816512    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.818471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.818447    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2294303a-f8f8-439d-869a-a16d8fe60a1a-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.818713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.818687    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.818945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.818926    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.819034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.819019    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2294303a-f8f8-439d-869a-a16d8fe60a1a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.827251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.827226    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/2294303a-f8f8-439d-869a-a16d8fe60a1a-kube-api-access-nknzg\") pod \"opendatahub-operator-controller-manager-689bcf9f9-nwc4n\" (UID: \"2294303a-f8f8-439d-869a-a16d8fe60a1a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:34.944483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:34.944394    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:35.249370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:35.249288    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"]
Nov 01 20:01:35.252684 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:35.252644    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2294303a_f8f8_439d_869a_a16d8fe60a1a.slice/crio-b4ce6bb2ae43c0174d6f14d96fc24a2ab0258cf6f705efe599996f90d8473bb3 WatchSource:0}: Error finding container b4ce6bb2ae43c0174d6f14d96fc24a2ab0258cf6f705efe599996f90d8473bb3: Status 404 returned error can't find the container with id b4ce6bb2ae43c0174d6f14d96fc24a2ab0258cf6f705efe599996f90d8473bb3
Nov 01 20:01:35.462211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:35.462172    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n" event={"ID":"2294303a-f8f8-439d-869a-a16d8fe60a1a","Type":"ContainerStarted","Data":"b4ce6bb2ae43c0174d6f14d96fc24a2ab0258cf6f705efe599996f90d8473bb3"}
Nov 01 20:01:39.483314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:39.483227    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n" event={"ID":"2294303a-f8f8-439d-869a-a16d8fe60a1a","Type":"ContainerStarted","Data":"42576e10b7df6758d22836c2efa2f3610d03e05ad2782caad0fb4641d06a0ec4"}
Nov 01 20:01:40.489328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:40.489296    5041 generic.go:356] "Generic (PLEG): container finished" podID="2294303a-f8f8-439d-869a-a16d8fe60a1a" containerID="42576e10b7df6758d22836c2efa2f3610d03e05ad2782caad0fb4641d06a0ec4" exitCode=0
Nov 01 20:01:40.489816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:40.489365    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n" event={"ID":"2294303a-f8f8-439d-869a-a16d8fe60a1a","Type":"ContainerDied","Data":"42576e10b7df6758d22836c2efa2f3610d03e05ad2782caad0fb4641d06a0ec4"}
Nov 01 20:01:41.495587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:41.495546    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n" event={"ID":"2294303a-f8f8-439d-869a-a16d8fe60a1a","Type":"ContainerStarted","Data":"c3a2e0871efa861bc1b0bb11c2d9740cd65e75769b6a01349fa96143d7fead64"}
Nov 01 20:01:41.496032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:41.495677    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:41.532563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:41.532506    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n" podStartSLOduration=2.763558238 podStartE2EDuration="7.532491757s" totalImagesPullingTime="4.313071234s" totalInitContainerRuntime="455.862285ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:41.529897982 +0000 UTC m=+189.250976658" watchObservedRunningTime="2026-11-01 20:01:41.532491757 +0000 UTC m=+189.253570430"
Nov 01 20:01:52.502716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:52.502683    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-nwc4n"
Nov 01 20:01:58.540201 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:01:58.540168    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:01:58.821387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.821317    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"]
Nov 01 20:01:58.825273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.825253    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:01:58.836199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.836171    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 20:01:58.837082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.837062    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 20:01:58.841274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.841247    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gqvhj\""
Nov 01 20:01:58.843840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.843814    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"]
Nov 01 20:01:58.921718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:58.921684    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q2x\" (UniqueName: \"kubernetes.io/projected/9d8742c6-60a8-4b53-b58e-ced648946d28-kube-api-access-97q2x\") pod \"authorino-operator-7b7bb4bc77-msx7k\" (UID: \"9d8742c6-60a8-4b53-b58e-ced648946d28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:01:59.022516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:59.022479    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q2x\" (UniqueName: \"kubernetes.io/projected/9d8742c6-60a8-4b53-b58e-ced648946d28-kube-api-access-97q2x\") pod \"authorino-operator-7b7bb4bc77-msx7k\" (UID: \"9d8742c6-60a8-4b53-b58e-ced648946d28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:01:59.031662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:59.031629    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q2x\" (UniqueName: \"kubernetes.io/projected/9d8742c6-60a8-4b53-b58e-ced648946d28-kube-api-access-97q2x\") pod \"authorino-operator-7b7bb4bc77-msx7k\" (UID: \"9d8742c6-60a8-4b53-b58e-ced648946d28\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:01:59.138455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:59.138372    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:01:59.414950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:59.414922    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"]
Nov 01 20:01:59.417441 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:01:59.417415    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8742c6_60a8_4b53_b58e_ced648946d28.slice/crio-210fc48350368d0c8c782ce01032b49fb259c4f821d564d926e2a3d82a8ee9fa WatchSource:0}: Error finding container 210fc48350368d0c8c782ce01032b49fb259c4f821d564d926e2a3d82a8ee9fa: Status 404 returned error can't find the container with id 210fc48350368d0c8c782ce01032b49fb259c4f821d564d926e2a3d82a8ee9fa
Nov 01 20:01:59.572990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:01:59.572952    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k" event={"ID":"9d8742c6-60a8-4b53-b58e-ced648946d28","Type":"ContainerStarted","Data":"210fc48350368d0c8c782ce01032b49fb259c4f821d564d926e2a3d82a8ee9fa"}
Nov 01 20:02:02.212983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.212946    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"]
Nov 01 20:02:02.215566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.215537    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:02.229444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.229417    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 20:02:02.229572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.229509    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bjgwq\""
Nov 01 20:02:02.229806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.229764    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"]
Nov 01 20:02:02.351603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.351571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4b4\" (UniqueName: \"kubernetes.io/projected/64f36934-c791-4d4a-b443-81e23ecb97ac-kube-api-access-jk4b4\") pod \"dns-operator-controller-manager-8557cbdb84-9hf26\" (UID: \"64f36934-c791-4d4a-b443-81e23ecb97ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:02.452096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.452058    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4b4\" (UniqueName: \"kubernetes.io/projected/64f36934-c791-4d4a-b443-81e23ecb97ac-kube-api-access-jk4b4\") pod \"dns-operator-controller-manager-8557cbdb84-9hf26\" (UID: \"64f36934-c791-4d4a-b443-81e23ecb97ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:02.480700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.480620    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4b4\" (UniqueName: \"kubernetes.io/projected/64f36934-c791-4d4a-b443-81e23ecb97ac-kube-api-access-jk4b4\") pod \"dns-operator-controller-manager-8557cbdb84-9hf26\" (UID: \"64f36934-c791-4d4a-b443-81e23ecb97ac\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:02.531218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.531189    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:02.615949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.606644    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"]
Nov 01 20:02:02.621510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.620637    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k" event={"ID":"9d8742c6-60a8-4b53-b58e-ced648946d28","Type":"ContainerStarted","Data":"4f913d478545e592d97ce9810f14ee1922386d90de87aa21c0b2d34bab4524ff"}
Nov 01 20:02:02.621510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.620757    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.621510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.621016    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:02:02.624854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.624584    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"]
Nov 01 20:02:02.634891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.630061    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 20:02:02.634891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.630228    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 20:02:02.634891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.630318    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jjbpg\""
Nov 01 20:02:02.657676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.657623    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k" podStartSLOduration=1.760302944 podStartE2EDuration="4.657607424s" totalImagesPullingTime="2.89730448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:02.654552438 +0000 UTC m=+210.375631117" watchObservedRunningTime="2026-11-01 20:02:02.657607424 +0000 UTC m=+210.378686100"
Nov 01 20:02:02.755415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.755339    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzr9\" (UniqueName: \"kubernetes.io/projected/57934152-4f00-4fa3-af31-2d825cbae99f-kube-api-access-tmzr9\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.755415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.755383    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57934152-4f00-4fa3-af31-2d825cbae99f-tls-certs\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.755597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.755558    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57934152-4f00-4fa3-af31-2d825cbae99f-tmp\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.842738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.842644    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"]
Nov 01 20:02:02.845097 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:02.845069    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f36934_c791_4d4a_b443_81e23ecb97ac.slice/crio-e38fade9dca8db9a235fee42f29d2ce471cd6afd186bd1b5965716ab8a66dd8a WatchSource:0}: Error finding container e38fade9dca8db9a235fee42f29d2ce471cd6afd186bd1b5965716ab8a66dd8a: Status 404 returned error can't find the container with id e38fade9dca8db9a235fee42f29d2ce471cd6afd186bd1b5965716ab8a66dd8a
Nov 01 20:02:02.856892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.856868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57934152-4f00-4fa3-af31-2d825cbae99f-tmp\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.856973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.856927    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzr9\" (UniqueName: \"kubernetes.io/projected/57934152-4f00-4fa3-af31-2d825cbae99f-kube-api-access-tmzr9\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.856973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.856953    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57934152-4f00-4fa3-af31-2d825cbae99f-tls-certs\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.859022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.858998    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57934152-4f00-4fa3-af31-2d825cbae99f-tmp\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.859220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.859203    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57934152-4f00-4fa3-af31-2d825cbae99f-tls-certs\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.873033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.873003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzr9\" (UniqueName: \"kubernetes.io/projected/57934152-4f00-4fa3-af31-2d825cbae99f-kube-api-access-tmzr9\") pod \"kube-auth-proxy-59c66b786f-x5rr8\" (UID: \"57934152-4f00-4fa3-af31-2d825cbae99f\") " pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:02.945848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:02.945814    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"
Nov 01 20:02:03.228802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:03.228762    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8"]
Nov 01 20:02:03.230732 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:03.230703    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57934152_4f00_4fa3_af31_2d825cbae99f.slice/crio-bdd114bd062969b89b1605a4ebf7d6d37b4cd2ab7520c777cfeac168b9d4f546 WatchSource:0}: Error finding container bdd114bd062969b89b1605a4ebf7d6d37b4cd2ab7520c777cfeac168b9d4f546: Status 404 returned error can't find the container with id bdd114bd062969b89b1605a4ebf7d6d37b4cd2ab7520c777cfeac168b9d4f546
Nov 01 20:02:03.626563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:03.626529    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26" event={"ID":"64f36934-c791-4d4a-b443-81e23ecb97ac","Type":"ContainerStarted","Data":"e38fade9dca8db9a235fee42f29d2ce471cd6afd186bd1b5965716ab8a66dd8a"}
Nov 01 20:02:03.627624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:03.627597    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8" event={"ID":"57934152-4f00-4fa3-af31-2d825cbae99f","Type":"ContainerStarted","Data":"bdd114bd062969b89b1605a4ebf7d6d37b4cd2ab7520c777cfeac168b9d4f546"}
Nov 01 20:02:05.683918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.683887    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"]
Nov 01 20:02:05.687740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.687715    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:05.697734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.697003    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4rc7d\""
Nov 01 20:02:05.698762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.698740    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"]
Nov 01 20:02:05.884592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.884542    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2z4t\" (UniqueName: \"kubernetes.io/projected/09f298b9-1369-4358-8649-2d3e06b15ba0-kube-api-access-k2z4t\") pod \"limitador-operator-controller-manager-5778bb4449-rrr99\" (UID: \"09f298b9-1369-4358-8649-2d3e06b15ba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:05.985345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.985321    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2z4t\" (UniqueName: \"kubernetes.io/projected/09f298b9-1369-4358-8649-2d3e06b15ba0-kube-api-access-k2z4t\") pod \"limitador-operator-controller-manager-5778bb4449-rrr99\" (UID: \"09f298b9-1369-4358-8649-2d3e06b15ba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:05.993496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:05.993470    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2z4t\" (UniqueName: \"kubernetes.io/projected/09f298b9-1369-4358-8649-2d3e06b15ba0-kube-api-access-k2z4t\") pod \"limitador-operator-controller-manager-5778bb4449-rrr99\" (UID: \"09f298b9-1369-4358-8649-2d3e06b15ba0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:06.006227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.006203    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:06.294718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.294689    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"]
Nov 01 20:02:06.297690 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:06.297664    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f298b9_1369_4358_8649_2d3e06b15ba0.slice/crio-8d4fe53108251c076c36d64b41eb1cc7f3282a2775e4cdd7198231208778f9c6 WatchSource:0}: Error finding container 8d4fe53108251c076c36d64b41eb1cc7f3282a2775e4cdd7198231208778f9c6: Status 404 returned error can't find the container with id 8d4fe53108251c076c36d64b41eb1cc7f3282a2775e4cdd7198231208778f9c6
Nov 01 20:02:06.645962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.645870    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99" event={"ID":"09f298b9-1369-4358-8649-2d3e06b15ba0","Type":"ContainerStarted","Data":"8d4fe53108251c076c36d64b41eb1cc7f3282a2775e4cdd7198231208778f9c6"}
Nov 01 20:02:06.647448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.647420    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8" event={"ID":"57934152-4f00-4fa3-af31-2d825cbae99f","Type":"ContainerStarted","Data":"8be07709577f990aa18557bff564e19381edd1fea8de3416daa0eddc47b5bdb2"}
Nov 01 20:02:06.649017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.648994    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26" event={"ID":"64f36934-c791-4d4a-b443-81e23ecb97ac","Type":"ContainerStarted","Data":"d788fee71a5cda50f1e8c146365ec1c52e1d2f9034bba3c32b11fd57c2a3ef9e"}
Nov 01 20:02:06.649132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.649117    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:06.685686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.685632    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59c66b786f-x5rr8" podStartSLOduration=2.233485081 podStartE2EDuration="4.685619724s" totalImagesPullingTime="2.452134643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:06.683832861 +0000 UTC m=+214.404911537" watchObservedRunningTime="2026-11-01 20:02:06.685619724 +0000 UTC m=+214.406698397"
Nov 01 20:02:06.717369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:06.717325    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26" podStartSLOduration=1.580824599 podStartE2EDuration="4.717311956s" totalImagesPullingTime="3.136487357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:06.713895714 +0000 UTC m=+214.434974390" watchObservedRunningTime="2026-11-01 20:02:06.717311956 +0000 UTC m=+214.438390632"
Nov 01 20:02:10.670014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:10.669973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99" event={"ID":"09f298b9-1369-4358-8649-2d3e06b15ba0","Type":"ContainerStarted","Data":"05dad3c54a9c0f91d52e2f73246da91b1c5dfea03364edad5392efbed5130043"}
Nov 01 20:02:10.670397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:10.670113    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:10.687732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:10.687685    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99" podStartSLOduration=1.876692372 podStartE2EDuration="5.687670455s" totalImagesPullingTime="3.810978083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:10.686458291 +0000 UTC m=+218.407536966" watchObservedRunningTime="2026-11-01 20:02:10.687670455 +0000 UTC m=+218.408749131"
Nov 01 20:02:14.638158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:14.638084    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:02:17.657368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:17.657336    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-9hf26"
Nov 01 20:02:21.676922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:21.676888    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rrr99"
Nov 01 20:02:28.539714 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:02:28.539677    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:02:51.578564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.578530    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"]
Nov 01 20:02:51.583189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.583165    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.593089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.592884    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"]
Nov 01 20:02:51.616582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.612705    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4pgmt\""
Nov 01 20:02:51.616582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.612695    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 20:02:51.616582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.612736    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 20:02:51.688830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.688799    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/74f17053-b4c9-40b0-8364-bbd0613232af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.689004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.688838    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/74f17053-b4c9-40b0-8364-bbd0613232af-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.689053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.688996    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9m78\" (UniqueName: \"kubernetes.io/projected/74f17053-b4c9-40b0-8364-bbd0613232af-kube-api-access-r9m78\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.789572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.789537    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9m78\" (UniqueName: \"kubernetes.io/projected/74f17053-b4c9-40b0-8364-bbd0613232af-kube-api-access-r9m78\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.789736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.789584    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/74f17053-b4c9-40b0-8364-bbd0613232af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.789736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.789610    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/74f17053-b4c9-40b0-8364-bbd0613232af-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.790301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.790277    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/74f17053-b4c9-40b0-8364-bbd0613232af-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.792140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.792113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/74f17053-b4c9-40b0-8364-bbd0613232af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.797339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.797318    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9m78\" (UniqueName: \"kubernetes.io/projected/74f17053-b4c9-40b0-8364-bbd0613232af-kube-api-access-r9m78\") pod \"kuadrant-console-plugin-66c4fb575d-5fx26\" (UID: \"74f17053-b4c9-40b0-8364-bbd0613232af\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:51.916572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:51.916482    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"
Nov 01 20:02:52.191393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.191369    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26"]
Nov 01 20:02:52.193573 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:52.193540    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74f17053_b4c9_40b0_8364_bbd0613232af.slice/crio-81be856b69a9ac84616fbb1aab4fea974c57595713d66a599970c02b1a86daa3 WatchSource:0}: Error finding container 81be856b69a9ac84616fbb1aab4fea974c57595713d66a599970c02b1a86daa3: Status 404 returned error can't find the container with id 81be856b69a9ac84616fbb1aab4fea974c57595713d66a599970c02b1a86daa3
Nov 01 20:02:52.372912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.372880    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:02:52.378245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.378224    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:52.385046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.385018    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pdfxz\""
Nov 01 20:02:52.386213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.386191    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:02:52.496199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.496125    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdf7\" (UniqueName: \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\") pod \"authorino-57f7f8659f-pkn6m\" (UID: \"56f556b6-76d2-4154-917e-8e10fd5cf8da\") " pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:52.596875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.596837    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdf7\" (UniqueName: \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\") pod \"authorino-57f7f8659f-pkn6m\" (UID: \"56f556b6-76d2-4154-917e-8e10fd5cf8da\") " pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:52.614645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.614617    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdf7\" (UniqueName: \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\") pod \"authorino-57f7f8659f-pkn6m\" (UID: \"56f556b6-76d2-4154-917e-8e10fd5cf8da\") " pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:52.692612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.692582    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:52.808562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.808532    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:02:52.812398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.812382    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:02:52.821340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.821314    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:02:52.872289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.872254    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26" event={"ID":"74f17053-b4c9-40b0-8364-bbd0613232af","Type":"ContainerStarted","Data":"81be856b69a9ac84616fbb1aab4fea974c57595713d66a599970c02b1a86daa3"}
Nov 01 20:02:52.972325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:52.972287    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:02:52.974512 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:52.974483    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56f556b6_76d2_4154_917e_8e10fd5cf8da.slice/crio-d714b4c6dfaec7e7a49b587b1602e25459917b0839832784cef3ee11f8a887f3 WatchSource:0}: Error finding container d714b4c6dfaec7e7a49b587b1602e25459917b0839832784cef3ee11f8a887f3: Status 404 returned error can't find the container with id d714b4c6dfaec7e7a49b587b1602e25459917b0839832784cef3ee11f8a887f3
Nov 01 20:02:53.000252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.000223    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rb8\" (UniqueName: \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\") pod \"authorino-56c6c9ddd5-qf7xf\" (UID: \"19fe1020-1d54-441a-8b28-3a5775664693\") " pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:02:53.101066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.100994    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rb8\" (UniqueName: \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\") pod \"authorino-56c6c9ddd5-qf7xf\" (UID: \"19fe1020-1d54-441a-8b28-3a5775664693\") " pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:02:53.115719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.115692    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rb8\" (UniqueName: \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\") pod \"authorino-56c6c9ddd5-qf7xf\" (UID: \"19fe1020-1d54-441a-8b28-3a5775664693\") " pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:02:53.125625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.125606    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:02:53.399557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.399480    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:02:53.407014 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:53.406983    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19fe1020_1d54_441a_8b28_3a5775664693.slice/crio-d7f7dbbb5a0186b4c8b9e024379acf16d5e491a643553de17c4a0aeb4177cd05 WatchSource:0}: Error finding container d7f7dbbb5a0186b4c8b9e024379acf16d5e491a643553de17c4a0aeb4177cd05: Status 404 returned error can't find the container with id d7f7dbbb5a0186b4c8b9e024379acf16d5e491a643553de17c4a0aeb4177cd05
Nov 01 20:02:53.879259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.879224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" event={"ID":"19fe1020-1d54-441a-8b28-3a5775664693","Type":"ContainerStarted","Data":"d7f7dbbb5a0186b4c8b9e024379acf16d5e491a643553de17c4a0aeb4177cd05"}
Nov 01 20:02:53.880481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:53.880459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" event={"ID":"56f556b6-76d2-4154-917e-8e10fd5cf8da","Type":"ContainerStarted","Data":"d714b4c6dfaec7e7a49b587b1602e25459917b0839832784cef3ee11f8a887f3"}
Nov 01 20:02:56.707915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.707879    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-cj2xb"]
Nov 01 20:02:56.719717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.719692    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.720011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.719979    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cj2xb"]
Nov 01 20:02:56.742251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.742196    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 01 20:02:56.742406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.742306    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 01 20:02:56.742406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.742382    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j8bld\""
Nov 01 20:02:56.751033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.750991    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 20:02:56.751033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.751024    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 20:02:56.843572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.843529    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-metrics-certs\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.843765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.843668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp76h\" (UniqueName: \"kubernetes.io/projected/a902a934-ca05-4ef4-a015-35ca5504e645-kube-api-access-sp76h\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.843765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.843721    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-cert\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.899248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.899208    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" event={"ID":"19fe1020-1d54-441a-8b28-3a5775664693","Type":"ContainerStarted","Data":"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"}
Nov 01 20:02:56.900927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.900899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" event={"ID":"56f556b6-76d2-4154-917e-8e10fd5cf8da","Type":"ContainerStarted","Data":"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"}
Nov 01 20:02:56.924013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.923952    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" podStartSLOduration=2.278313121 podStartE2EDuration="4.923932916s" totalImagesPullingTime="2.645619795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:56.921603479 +0000 UTC m=+264.642682155" watchObservedRunningTime="2026-11-01 20:02:56.923932916 +0000 UTC m=+264.645011593"
Nov 01 20:02:56.944599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.944562    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-metrics-certs\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.944810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.944709    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp76h\" (UniqueName: \"kubernetes.io/projected/a902a934-ca05-4ef4-a015-35ca5504e645-kube-api-access-sp76h\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.944810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.944744    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-cert\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.947763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.947737    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-cert\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.947948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.947921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a902a934-ca05-4ef4-a015-35ca5504e645-metrics-certs\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:56.948967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.948923    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" podStartSLOduration=1.864740182 podStartE2EDuration="4.948910742s" totalImagesPullingTime="3.08417056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:56.947948947 +0000 UTC m=+264.669027625" watchObservedRunningTime="2026-11-01 20:02:56.948910742 +0000 UTC m=+264.669989417"
Nov 01 20:02:56.973110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:56.973039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp76h\" (UniqueName: \"kubernetes.io/projected/a902a934-ca05-4ef4-a015-35ca5504e645-kube-api-access-sp76h\") pod \"odh-model-controller-769875db56-cj2xb\" (UID: \"a902a934-ca05-4ef4-a015-35ca5504e645\") " pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:57.009896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:57.009854    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:02:57.040035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:57.040001    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:02:57.366048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:57.366022    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-cj2xb"]
Nov 01 20:02:57.369021 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:02:57.368983    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda902a934_ca05_4ef4_a015_35ca5504e645.slice/crio-f03aecc229b8d58722f1926169f8a13037275a0f2aa6360d41ccd90cb378129d WatchSource:0}: Error finding container f03aecc229b8d58722f1926169f8a13037275a0f2aa6360d41ccd90cb378129d: Status 404 returned error can't find the container with id f03aecc229b8d58722f1926169f8a13037275a0f2aa6360d41ccd90cb378129d
Nov 01 20:02:57.908663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:57.908631    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cj2xb" event={"ID":"a902a934-ca05-4ef4-a015-35ca5504e645","Type":"ContainerStarted","Data":"f03aecc229b8d58722f1926169f8a13037275a0f2aa6360d41ccd90cb378129d"}
Nov 01 20:02:58.554503 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:02:58.554468    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:02:58.915900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:58.915815    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26" event={"ID":"74f17053-b4c9-40b0-8364-bbd0613232af","Type":"ContainerStarted","Data":"cb1b8aef15df2cc14dce38db73b19ae745a6ece66ae06f7727d46bbe562525ea"}
Nov 01 20:02:58.916357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:58.915971    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" podUID="56f556b6-76d2-4154-917e-8e10fd5cf8da" containerName="authorino" containerID="cri-o://71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d" gracePeriod=30
Nov 01 20:02:58.931056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:58.930962    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5fx26" podStartSLOduration=2.287310037 podStartE2EDuration="7.930944971s" totalImagesPullingTime="5.643634934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:58.930681459 +0000 UTC m=+266.651760136" watchObservedRunningTime="2026-11-01 20:02:58.930944971 +0000 UTC m=+266.652023647"
Nov 01 20:02:59.192189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.192160    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:59.265855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.265825    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\" (UniqueName: \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\") pod \"56f556b6-76d2-4154-917e-8e10fd5cf8da\" (UID: \"56f556b6-76d2-4154-917e-8e10fd5cf8da\") "
Nov 01 20:02:59.268439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.268409    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7" pod "56f556b6-76d2-4154-917e-8e10fd5cf8da" (UID: "56f556b6-76d2-4154-917e-8e10fd5cf8da"). InnerVolumeSpecName "kube-api-access-jvdf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:02:59.366891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.366857    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdf7\" (UniqueName: \"kubernetes.io/projected/56f556b6-76d2-4154-917e-8e10fd5cf8da-kube-api-access-jvdf7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:02:59.922200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.922165    5041 generic.go:356] "Generic (PLEG): container finished" podID="56f556b6-76d2-4154-917e-8e10fd5cf8da" containerID="71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d" exitCode=0
Nov 01 20:02:59.922621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.922201    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" event={"ID":"56f556b6-76d2-4154-917e-8e10fd5cf8da","Type":"ContainerDied","Data":"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"}
Nov 01 20:02:59.922621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.922228    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pkn6m"
Nov 01 20:02:59.922621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.922261    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pkn6m" event={"ID":"56f556b6-76d2-4154-917e-8e10fd5cf8da","Type":"ContainerDied","Data":"d714b4c6dfaec7e7a49b587b1602e25459917b0839832784cef3ee11f8a887f3"}
Nov 01 20:02:59.922621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.922286    5041 scope.go:122] "RemoveContainer" containerID="71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"
Nov 01 20:02:59.932209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.932186    5041 scope.go:122] "RemoveContainer" containerID="71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"
Nov 01 20:02:59.932827 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:02:59.932490    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d\": container with ID starting with 71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d not found: ID does not exist" containerID="71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"
Nov 01 20:02:59.932827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.932522    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d"} err="failed to get container status \"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d\": rpc error: code = NotFound desc = could not find container \"71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d\": container with ID starting with 71eaceefb645e29eb63fd7dd4ba6716596a20ad34979adedc23d61914e0a698d not found: ID does not exist"
Nov 01 20:02:59.946245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.946218    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:02:59.949732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:02:59.949711    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pkn6m"]
Nov 01 20:03:00.426761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:00.426728    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f556b6-76d2-4154-917e-8e10fd5cf8da" path="/var/lib/kubelet/pods/56f556b6-76d2-4154-917e-8e10fd5cf8da/volumes"
Nov 01 20:03:00.929198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:00.929160    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-cj2xb" event={"ID":"a902a934-ca05-4ef4-a015-35ca5504e645","Type":"ContainerStarted","Data":"421e2139b8e649f6780f7253164a3a7249a9d2c0860ffb7b2d6f905f742fc39e"}
Nov 01 20:03:00.929577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:00.929331    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:03:00.945700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:00.945651    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-cj2xb" podStartSLOduration=2.07914619 podStartE2EDuration="4.945635873s" totalImagesPullingTime="2.866489683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:00.943953479 +0000 UTC m=+268.665032155" watchObservedRunningTime="2026-11-01 20:03:00.945635873 +0000 UTC m=+268.666714552"
Nov 01 20:03:11.936398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:11.936369    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-cj2xb"
Nov 01 20:03:28.540200 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:03:28.540161    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:03:30.452875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.452842    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:30.453251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.453214    5041 cpu_manager.go:414] "containerMap: removing container" podUID="56f556b6-76d2-4154-917e-8e10fd5cf8da" containerName="authorino"
Nov 01 20:03:30.453251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.453225    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f556b6-76d2-4154-917e-8e10fd5cf8da" containerName="authorino"
Nov 01 20:03:30.453327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.453306    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f556b6-76d2-4154-917e-8e10fd5cf8da" containerName="authorino"
Nov 01 20:03:30.470802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.470744    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:30.470966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.470915    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.476095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.476067    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xqt97\""
Nov 01 20:03:30.476095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.476091    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 20:03:30.476312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.476164    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 20:03:30.476312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.476241    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 20:03:30.532009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.531974    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.532159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.532016    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.532159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.532145    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6ts\" (UniqueName: \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.633171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.633128    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6ts\" (UniqueName: \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.633372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.633211    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.633372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.633246    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.635874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.635843    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.635994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.635843    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.640472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.640451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6ts\" (UniqueName: \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\") pod \"maas-controller-7f9f878d4b-fwg8j\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") " pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.785245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.785215    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:30.914620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.914587    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:03:30.917287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.917271    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:30.927593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:30.927567    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:03:31.036598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.036517    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sbh\" (UniqueName: \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.036598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.036572    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.036814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.036707    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.056489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.056467    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:31.059075 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:03:31.059042    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f97ec51_8dc9_47ed_8ef4_23628d39fa0f.slice/crio-adc7995bef33ee7be989dbd67239d004e091ffded85069575d204ec98bfa4fa3 WatchSource:0}: Error finding container adc7995bef33ee7be989dbd67239d004e091ffded85069575d204ec98bfa4fa3: Status 404 returned error can't find the container with id adc7995bef33ee7be989dbd67239d004e091ffded85069575d204ec98bfa4fa3
Nov 01 20:03:31.078441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.078416    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" event={"ID":"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f","Type":"ContainerStarted","Data":"adc7995bef33ee7be989dbd67239d004e091ffded85069575d204ec98bfa4fa3"}
Nov 01 20:03:31.137850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.137815    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.138006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.137861    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sbh\" (UniqueName: \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.138006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.137884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.140366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.140338    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.140487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.140368    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.145529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.145503    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sbh\" (UniqueName: \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\") pod \"maas-controller-7f997f45fb-9tqcn\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") " pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.230807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.230741    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:31.505768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:31.505722    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:03:31.507815 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:03:31.507771    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc296fd5a_f848_401e_990a_f1cdb60c5ccc.slice/crio-bb32c631c5cd52492a679668aa40ac3ce029387841a01cf290435f8fd4163d5e WatchSource:0}: Error finding container bb32c631c5cd52492a679668aa40ac3ce029387841a01cf290435f8fd4163d5e: Status 404 returned error can't find the container with id bb32c631c5cd52492a679668aa40ac3ce029387841a01cf290435f8fd4163d5e
Nov 01 20:03:32.086541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:32.086508    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" event={"ID":"c296fd5a-f848-401e-990a-f1cdb60c5ccc","Type":"ContainerStarted","Data":"bb32c631c5cd52492a679668aa40ac3ce029387841a01cf290435f8fd4163d5e"}
Nov 01 20:03:32.471439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:32.471413    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:03:32.472965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:32.472944    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:03:35.102273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:35.102247    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 20:03:36.108412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.108370    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" event={"ID":"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f","Type":"ContainerStarted","Data":"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"}
Nov 01 20:03:36.108876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.108566    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:36.110047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.110021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" event={"ID":"c296fd5a-f848-401e-990a-f1cdb60c5ccc","Type":"ContainerStarted","Data":"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"}
Nov 01 20:03:36.110151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.110071    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:36.128302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.128258    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" podStartSLOduration=2.089431899 podStartE2EDuration="6.128244688s" totalImagesPullingTime="4.038812789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:36.127025203 +0000 UTC m=+303.848103879" watchObservedRunningTime="2026-11-01 20:03:36.128244688 +0000 UTC m=+303.849323360"
Nov 01 20:03:36.144347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:36.144302    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" podStartSLOduration=2.546763845 podStartE2EDuration="6.144288536s" totalImagesPullingTime="3.597524691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:36.144006054 +0000 UTC m=+303.865084731" watchObservedRunningTime="2026-11-01 20:03:36.144288536 +0000 UTC m=+303.865367213"
Nov 01 20:03:47.117405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.117333    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:03:47.117764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.117413    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:47.183263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.183224    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:47.183490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.183442    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" podUID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" containerName="manager" containerID="cri-o://eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50" gracePeriod=30
Nov 01 20:03:47.430272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.430246    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:47.597106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.597075    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\") pod \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") "
Nov 01 20:03:47.597290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.597131    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\") pod \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") "
Nov 01 20:03:47.597290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.597242    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\" (UniqueName: \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\") pod \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\" (UID: \"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f\") "
Nov 01 20:03:47.599318 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.599289    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs" pod "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" (UID: "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:47.599418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.599318    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert" pod "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" (UID: "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:03:47.599418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.599353    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts" pod "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" (UID: "6f97ec51-8dc9-47ed-8ef4-23628d39fa0f"). InnerVolumeSpecName "kube-api-access-7v6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:03:47.698433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.698345    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-webhook-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:03:47.698433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.698397    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6ts\" (UniqueName: \"kubernetes.io/projected/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-kube-api-access-7v6ts\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:03:47.698433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:47.698408    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f-metrics-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:03:48.171338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.171297    5041 generic.go:356] "Generic (PLEG): container finished" podID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" containerID="eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50" exitCode=0
Nov 01 20:03:48.171798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.171344    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j"
Nov 01 20:03:48.171798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.171358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" event={"ID":"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f","Type":"ContainerDied","Data":"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"}
Nov 01 20:03:48.171798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.171412    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9f878d4b-fwg8j" event={"ID":"6f97ec51-8dc9-47ed-8ef4-23628d39fa0f","Type":"ContainerDied","Data":"adc7995bef33ee7be989dbd67239d004e091ffded85069575d204ec98bfa4fa3"}
Nov 01 20:03:48.171798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.171442    5041 scope.go:122] "RemoveContainer" containerID="eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"
Nov 01 20:03:48.180608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.180589    5041 scope.go:122] "RemoveContainer" containerID="eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"
Nov 01 20:03:48.180892 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:03:48.180872    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50\": container with ID starting with eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50 not found: ID does not exist" containerID="eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"
Nov 01 20:03:48.180963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.180903    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50"} err="failed to get container status \"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50\": rpc error: code = NotFound desc = could not find container \"eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50\": container with ID starting with eb7a8a2470484e80059b45753934052df6b39ae06da259522d25251c5faa5c50 not found: ID does not exist"
Nov 01 20:03:48.192091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.192066    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:48.195426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.195405    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9f878d4b-fwg8j"]
Nov 01 20:03:48.428133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:03:48.428052    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" path="/var/lib/kubelet/pods/6f97ec51-8dc9-47ed-8ef4-23628d39fa0f/volumes"
Nov 01 20:03:58.540548 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:03:58.540504    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:04:19.688394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.688361    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-xj4fd"]
Nov 01 20:04:19.688872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.688769    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" containerName="manager"
Nov 01 20:04:19.688872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.688799    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" containerName="manager"
Nov 01 20:04:19.688872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.688864    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f97ec51-8dc9-47ed-8ef4-23628d39fa0f" containerName="manager"
Nov 01 20:04:19.691851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.691835    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.697801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.697754    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 20:04:19.699238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.699216    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-xj4fd"]
Nov 01 20:04:19.729155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.729122    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-xj4fd"]
Nov 01 20:04:19.729338 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:04:19.729321    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qc6n5 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-xj4fd" podUID="72a51aef-7386-4bbf-80ad-45c1ddd1ae43"
Nov 01 20:04:19.742856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.742828    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6n5\" (UniqueName: \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.742997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.742951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.753864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.753841    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:19.757288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.757271    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.765439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.765412    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:19.843418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.843389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.843418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.843423    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.843643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.843481    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6n5\" (UniqueName: \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.843643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.843525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxnh\" (UniqueName: \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.845984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.845964    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.850637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.850616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6n5\" (UniqueName: \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\") pod \"authorino-6dd76875b5-xj4fd\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") " pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:19.944688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.944596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.944906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.944693    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxnh\" (UniqueName: \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.947051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.947029    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:19.952373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:19.952352    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxnh\" (UniqueName: \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\") pod \"authorino-6889cbd8c4-csx4b\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") " pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:20.045501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.045465    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:20.045703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.045685    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:20.062963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.062937    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:04:20.066849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.066828    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.072913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.072885    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:04:20.146231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.146197    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.146395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.146250    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67hx\" (UniqueName: \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.247028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.246931    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67hx\" (UniqueName: \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.247212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.247040    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.249444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.249419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.255166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.255136    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67hx\" (UniqueName: \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\") pod \"authorino-564698f58b-mv8bw\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") " pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.323756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.323730    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:20.326179 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:04:20.326148    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bbffc50_4c46_4a68_ae19_a11c84e56071.slice/crio-813bd7f37a61b27249e44cdfadb56b71d9507f04d254f8938c5f8ca6337383d2 WatchSource:0}: Error finding container 813bd7f37a61b27249e44cdfadb56b71d9507f04d254f8938c5f8ca6337383d2: Status 404 returned error can't find the container with id 813bd7f37a61b27249e44cdfadb56b71d9507f04d254f8938c5f8ca6337383d2
Nov 01 20:04:20.327576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.327556    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:04:20.339890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.339845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" event={"ID":"1bbffc50-4c46-4a68-ae19-a11c84e56071","Type":"ContainerStarted","Data":"813bd7f37a61b27249e44cdfadb56b71d9507f04d254f8938c5f8ca6337383d2"}
Nov 01 20:04:20.340022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.339951    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:20.348037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.348015    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:20.398010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.397979    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:04:20.448060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.447933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\" (UniqueName: \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\") pod \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") "
Nov 01 20:04:20.448060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.448021    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\" (UniqueName: \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\") pod \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\" (UID: \"72a51aef-7386-4bbf-80ad-45c1ddd1ae43\") "
Nov 01 20:04:20.450990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.450485    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5" pod "72a51aef-7386-4bbf-80ad-45c1ddd1ae43" (UID: "72a51aef-7386-4bbf-80ad-45c1ddd1ae43"). InnerVolumeSpecName "kube-api-access-qc6n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:04:20.450990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.450622    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert" pod "72a51aef-7386-4bbf-80ad-45c1ddd1ae43" (UID: "72a51aef-7386-4bbf-80ad-45c1ddd1ae43"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:04:20.549342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.549309    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6n5\" (UniqueName: \"kubernetes.io/projected/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-kube-api-access-qc6n5\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:04:20.549342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.549341    5041 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/72a51aef-7386-4bbf-80ad-45c1ddd1ae43-tls-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:04:20.673308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:20.673282    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:04:20.674453 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:04:20.674424    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddabfeb27_eecb_4658_a6fc_cb32a53300da.slice/crio-1fd29aa891687f5243b94e8b5e2c3fd59a6fe0607e9471c2021cf460075f18a7 WatchSource:0}: Error finding container 1fd29aa891687f5243b94e8b5e2c3fd59a6fe0607e9471c2021cf460075f18a7: Status 404 returned error can't find the container with id 1fd29aa891687f5243b94e8b5e2c3fd59a6fe0607e9471c2021cf460075f18a7
Nov 01 20:04:21.346190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:21.346157    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-xj4fd"
Nov 01 20:04:21.346190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:21.346173    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564698f58b-mv8bw" event={"ID":"dabfeb27-eecb-4658-a6fc-cb32a53300da","Type":"ContainerStarted","Data":"1fd29aa891687f5243b94e8b5e2c3fd59a6fe0607e9471c2021cf460075f18a7"}
Nov 01 20:04:21.374284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:21.374256    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-xj4fd"]
Nov 01 20:04:21.378924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:21.378900    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-xj4fd"]
Nov 01 20:04:22.354284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.354242    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" event={"ID":"1bbffc50-4c46-4a68-ae19-a11c84e56071","Type":"ContainerStarted","Data":"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"}
Nov 01 20:04:22.354742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.354316    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" podUID="1bbffc50-4c46-4a68-ae19-a11c84e56071" containerName="authorino" containerID="cri-o://6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98" gracePeriod=30
Nov 01 20:04:22.375163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.375107    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" podStartSLOduration=1.883333049 podStartE2EDuration="3.375088402s" totalImagesPullingTime="1.491755353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:04:22.371344384 +0000 UTC m=+350.092423060" watchObservedRunningTime="2026-11-01 20:04:22.375088402 +0000 UTC m=+350.096167079"
Nov 01 20:04:22.426601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.426567    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a51aef-7386-4bbf-80ad-45c1ddd1ae43" path="/var/lib/kubelet/pods/72a51aef-7386-4bbf-80ad-45c1ddd1ae43/volumes"
Nov 01 20:04:22.613927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.613863    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:22.668763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.668727    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\" (UniqueName: \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\") pod \"1bbffc50-4c46-4a68-ae19-a11c84e56071\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") "
Nov 01 20:04:22.668948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.668806    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\" (UniqueName: \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\") pod \"1bbffc50-4c46-4a68-ae19-a11c84e56071\" (UID: \"1bbffc50-4c46-4a68-ae19-a11c84e56071\") "
Nov 01 20:04:22.670716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.670688    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh" pod "1bbffc50-4c46-4a68-ae19-a11c84e56071" (UID: "1bbffc50-4c46-4a68-ae19-a11c84e56071"). InnerVolumeSpecName "kube-api-access-bmxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:04:22.679481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.679453    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert" pod "1bbffc50-4c46-4a68-ae19-a11c84e56071" (UID: "1bbffc50-4c46-4a68-ae19-a11c84e56071"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:04:22.769920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.769885    5041 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1bbffc50-4c46-4a68-ae19-a11c84e56071-tls-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:04:22.769920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:22.769916    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxnh\" (UniqueName: \"kubernetes.io/projected/1bbffc50-4c46-4a68-ae19-a11c84e56071-kube-api-access-bmxnh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:04:23.367865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.367757    5041 generic.go:356] "Generic (PLEG): container finished" podID="1bbffc50-4c46-4a68-ae19-a11c84e56071" containerID="6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98" exitCode=0
Nov 01 20:04:23.368335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.367963    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" event={"ID":"1bbffc50-4c46-4a68-ae19-a11c84e56071","Type":"ContainerDied","Data":"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"}
Nov 01 20:04:23.368335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.367995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-csx4b" event={"ID":"1bbffc50-4c46-4a68-ae19-a11c84e56071","Type":"ContainerDied","Data":"813bd7f37a61b27249e44cdfadb56b71d9507f04d254f8938c5f8ca6337383d2"}
Nov 01 20:04:23.368335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.368013    5041 scope.go:122] "RemoveContainer" containerID="6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"
Nov 01 20:04:23.368335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.368139    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-csx4b"
Nov 01 20:04:23.377586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.377549    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564698f58b-mv8bw" event={"ID":"dabfeb27-eecb-4658-a6fc-cb32a53300da","Type":"ContainerStarted","Data":"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"}
Nov 01 20:04:23.386912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.386886    5041 scope.go:122] "RemoveContainer" containerID="6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"
Nov 01 20:04:23.387270 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:04:23.387244    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98\": container with ID starting with 6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98 not found: ID does not exist" containerID="6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"
Nov 01 20:04:23.387387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.387285    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98"} err="failed to get container status \"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98\": rpc error: code = NotFound desc = could not find container \"6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98\": container with ID starting with 6bc625fea30ac3dc841a5710f41b864d8865a4413790525c605e3e5d56d44e98 not found: ID does not exist"
Nov 01 20:04:23.427639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.423193    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-564698f58b-mv8bw" podStartSLOduration=1.075846553 podStartE2EDuration="3.423173617s" totalImagesPullingTime="2.347327064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:04:23.408429847 +0000 UTC m=+351.129508523" watchObservedRunningTime="2026-11-01 20:04:23.423173617 +0000 UTC m=+351.144252293"
Nov 01 20:04:23.427639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.425763    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:23.433284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.428844    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-csx4b"]
Nov 01 20:04:23.434275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.433675    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:04:23.434275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.433904    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" podUID="19fe1020-1d54-441a-8b28-3a5775664693" containerName="authorino" containerID="cri-o://f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e" gracePeriod=30
Nov 01 20:04:23.692251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.692230    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:04:23.779161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.779129    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\" (UniqueName: \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\") pod \"19fe1020-1d54-441a-8b28-3a5775664693\" (UID: \"19fe1020-1d54-441a-8b28-3a5775664693\") "
Nov 01 20:04:23.781179 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.781158    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8" pod "19fe1020-1d54-441a-8b28-3a5775664693" (UID: "19fe1020-1d54-441a-8b28-3a5775664693"). InnerVolumeSpecName "kube-api-access-v8rb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:04:23.880350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:23.880304    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rb8\" (UniqueName: \"kubernetes.io/projected/19fe1020-1d54-441a-8b28-3a5775664693-kube-api-access-v8rb8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:04:24.383755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.383665    5041 generic.go:356] "Generic (PLEG): container finished" podID="19fe1020-1d54-441a-8b28-3a5775664693" containerID="f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e" exitCode=0
Nov 01 20:04:24.383755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.383712    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf"
Nov 01 20:04:24.383755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.383745    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" event={"ID":"19fe1020-1d54-441a-8b28-3a5775664693","Type":"ContainerDied","Data":"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"}
Nov 01 20:04:24.384374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.383807    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qf7xf" event={"ID":"19fe1020-1d54-441a-8b28-3a5775664693","Type":"ContainerDied","Data":"d7f7dbbb5a0186b4c8b9e024379acf16d5e491a643553de17c4a0aeb4177cd05"}
Nov 01 20:04:24.384374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.383835    5041 scope.go:122] "RemoveContainer" containerID="f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"
Nov 01 20:04:24.399913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.399889    5041 scope.go:122] "RemoveContainer" containerID="f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"
Nov 01 20:04:24.400191 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:04:24.400171    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e\": container with ID starting with f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e not found: ID does not exist" containerID="f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"
Nov 01 20:04:24.400232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.400200    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e"} err="failed to get container status \"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e\": rpc error: code = NotFound desc = could not find container \"f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e\": container with ID starting with f2fe83a9d235993fc54411fc0494309825b494966a6fc4f3c1cc7e888003914e not found: ID does not exist"
Nov 01 20:04:24.408235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.408213    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:04:24.410589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.410566    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qf7xf"]
Nov 01 20:04:24.427042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.427017    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fe1020-1d54-441a-8b28-3a5775664693" path="/var/lib/kubelet/pods/19fe1020-1d54-441a-8b28-3a5775664693/volumes"
Nov 01 20:04:24.427361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:24.427349    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbffc50-4c46-4a68-ae19-a11c84e56071" path="/var/lib/kubelet/pods/1bbffc50-4c46-4a68-ae19-a11c84e56071/volumes"
Nov 01 20:04:28.540848 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:04:28.540810    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:04:58.176511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176476    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176853    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1bbffc50-4c46-4a68-ae19-a11c84e56071" containerName="authorino"
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176864    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbffc50-4c46-4a68-ae19-a11c84e56071" containerName="authorino"
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176890    5041 cpu_manager.go:414] "containerMap: removing container" podUID="19fe1020-1d54-441a-8b28-3a5775664693" containerName="authorino"
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176897    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fe1020-1d54-441a-8b28-3a5775664693" containerName="authorino"
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176963    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbffc50-4c46-4a68-ae19-a11c84e56071" containerName="authorino"
Nov 01 20:04:58.176986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.176975    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fe1020-1d54-441a-8b28-3a5775664693" containerName="authorino"
Nov 01 20:04:58.180233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.180214    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.191067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.191036    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:04:58.195652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.195624    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 01 20:04:58.196501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.196478    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z2j2w\""
Nov 01 20:04:58.196610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.196546    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 01 20:04:58.269227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.269189    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.269379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.269263    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsslb\" (UniqueName: \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.370334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.370297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsslb\" (UniqueName: \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.370492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.370373    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.372808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.372765    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.378134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.378109    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsslb\" (UniqueName: \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\") pod \"ai-gateway-controller-7d85675f98-rgmfq\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") " pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.498604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.498526    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:04:58.541899 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:04:58.541861    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:04:58.773762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:58.773734    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:04:58.775703 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:04:58.775676    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27d37b4_1d54_4b46_9d4f_50b0af39744d.slice/crio-5b553f70e5f93afc9c22d8e499940a93aca0dc50d4a793b307c20f5c68c4f498 WatchSource:0}: Error finding container 5b553f70e5f93afc9c22d8e499940a93aca0dc50d4a793b307c20f5c68c4f498: Status 404 returned error can't find the container with id 5b553f70e5f93afc9c22d8e499940a93aca0dc50d4a793b307c20f5c68c4f498
Nov 01 20:04:59.563200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:04:59.563168    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" event={"ID":"e27d37b4-1d54-4b46-9d4f-50b0af39744d","Type":"ContainerStarted","Data":"5b553f70e5f93afc9c22d8e499940a93aca0dc50d4a793b307c20f5c68c4f498"}
Nov 01 20:05:02.580372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:02.580332    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" event={"ID":"e27d37b4-1d54-4b46-9d4f-50b0af39744d","Type":"ContainerStarted","Data":"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"}
Nov 01 20:05:02.580831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:02.580466    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:05:02.599493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:02.599445    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" podStartSLOduration=1.8648145029999998 podStartE2EDuration="4.599430447s" totalImagesPullingTime="2.734615944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:02.598222909 +0000 UTC m=+390.319301601" watchObservedRunningTime="2026-11-01 20:05:02.599430447 +0000 UTC m=+390.320509130"
Nov 01 20:05:03.308000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.307969    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:05:03.311632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.311615    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.320955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.320728    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:05:03.320955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.320807    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vzlvj\""
Nov 01 20:05:03.321158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.320965    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 01 20:05:03.332135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.332108    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:05:03.335558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.335534    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.347603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.347572    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:05:03.411951 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.411917    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.412117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.411979    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzgc\" (UniqueName: \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.512684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.512642    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.512926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.512838    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzgc\" (UniqueName: \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.513007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.512977    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prf9d\" (UniqueName: \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.513117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.513099    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.513458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.513438    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.520341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.520311    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzgc\" (UniqueName: \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\") pod \"payload-processing-55fbbbf9dc-r5wf2\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.614543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.614452    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prf9d\" (UniqueName: \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.614543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.614513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.615283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.615263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.622319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.622294    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prf9d\" (UniqueName: \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\") pod \"payload-pre-processing-7b98bb678c-wwqfh\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.626091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.626070    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:03.650529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.650493    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:03.905722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.905698    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:05:03.908179 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:03.908150    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a3b40_cb74_4cbc_824f_d1dc7bacf92f.slice/crio-daf50433a6aee26319b81ee10dd27fd189cea14a31a599c3d23312920d0d6738 WatchSource:0}: Error finding container daf50433a6aee26319b81ee10dd27fd189cea14a31a599c3d23312920d0d6738: Status 404 returned error can't find the container with id daf50433a6aee26319b81ee10dd27fd189cea14a31a599c3d23312920d0d6738
Nov 01 20:05:03.935031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:03.935009    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:05:03.937059 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:03.937035    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49da114b_50b0_4852_a9a8_8c36be0ac8d4.slice/crio-6954c5f688faaf3bcab46e884c9cb08ceb3d6448e4db69585e51892012ec5c3b WatchSource:0}: Error finding container 6954c5f688faaf3bcab46e884c9cb08ceb3d6448e4db69585e51892012ec5c3b: Status 404 returned error can't find the container with id 6954c5f688faaf3bcab46e884c9cb08ceb3d6448e4db69585e51892012ec5c3b
Nov 01 20:05:04.594957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:04.594916    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" event={"ID":"49da114b-50b0-4852-a9a8-8c36be0ac8d4","Type":"ContainerStarted","Data":"6954c5f688faaf3bcab46e884c9cb08ceb3d6448e4db69585e51892012ec5c3b"}
Nov 01 20:05:04.597030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:04.596995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" event={"ID":"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f","Type":"ContainerStarted","Data":"daf50433a6aee26319b81ee10dd27fd189cea14a31a599c3d23312920d0d6738"}
Nov 01 20:05:06.608902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.608866    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" event={"ID":"49da114b-50b0-4852-a9a8-8c36be0ac8d4","Type":"ContainerStarted","Data":"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"}
Nov 01 20:05:06.609290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.609004    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:06.610361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.610335    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" event={"ID":"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f","Type":"ContainerStarted","Data":"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"}
Nov 01 20:05:06.610474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.610376    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:06.624828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.624763    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" podStartSLOduration=1.327651742 podStartE2EDuration="3.6247522s" totalImagesPullingTime="2.297100458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:06.623484679 +0000 UTC m=+394.344563349" watchObservedRunningTime="2026-11-01 20:05:06.6247522 +0000 UTC m=+394.345830876"
Nov 01 20:05:06.640506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:06.640463    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" podStartSLOduration=1.075683404 podStartE2EDuration="3.640450979s" totalImagesPullingTime="2.564767575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:06.637991508 +0000 UTC m=+394.359070184" watchObservedRunningTime="2026-11-01 20:05:06.640450979 +0000 UTC m=+394.361529656"
Nov 01 20:05:13.589461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.589388    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:05:13.827492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.827460    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-f9d557c9-9x2z8"]
Nov 01 20:05:13.831578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.831557    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:13.838281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.838247    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-f9d557c9-9x2z8"]
Nov 01 20:05:13.898133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.898054    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klw2t\" (UniqueName: \"kubernetes.io/projected/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-kube-api-access-klw2t\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:13.898133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.898114    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-metrics-certs\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:13.998841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.998804    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klw2t\" (UniqueName: \"kubernetes.io/projected/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-kube-api-access-klw2t\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:13.999010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:13.998863    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-metrics-certs\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:14.001173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:14.001154    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-metrics-certs\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:14.006638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:14.006612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klw2t\" (UniqueName: \"kubernetes.io/projected/95343c71-5cc5-4b8d-a194-6eaa3433eb6f-kube-api-access-klw2t\") pod \"ai-gateway-controller-f9d557c9-9x2z8\" (UID: \"95343c71-5cc5-4b8d-a194-6eaa3433eb6f\") " pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:14.146340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:14.146300    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:14.419912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:14.419887    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-f9d557c9-9x2z8"]
Nov 01 20:05:14.422247 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:14.422221    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95343c71_5cc5_4b8d_a194_6eaa3433eb6f.slice/crio-498fb87ca6153d66264f43c178f83662930d648a155efc9716bb8fee8f2a3645 WatchSource:0}: Error finding container 498fb87ca6153d66264f43c178f83662930d648a155efc9716bb8fee8f2a3645: Status 404 returned error can't find the container with id 498fb87ca6153d66264f43c178f83662930d648a155efc9716bb8fee8f2a3645
Nov 01 20:05:14.656072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:14.656033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8" event={"ID":"95343c71-5cc5-4b8d-a194-6eaa3433eb6f","Type":"ContainerStarted","Data":"498fb87ca6153d66264f43c178f83662930d648a155efc9716bb8fee8f2a3645"}
Nov 01 20:05:15.662182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:15.662143    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8" event={"ID":"95343c71-5cc5-4b8d-a194-6eaa3433eb6f","Type":"ContainerStarted","Data":"035d2a74f7e63fed4f15425da7fb3fe9c98a2b27cd316a796e72dd283375341f"}
Nov 01 20:05:15.662653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:15.662285    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:15.693885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:15.693839    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8" podStartSLOduration=2.212444032 podStartE2EDuration="2.693826404s" totalImagesPullingTime="481.382372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:15.691637794 +0000 UTC m=+403.412716480" watchObservedRunningTime="2026-11-01 20:05:15.693826404 +0000 UTC m=+403.414905077"
Nov 01 20:05:17.618368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:17.618339    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:05:17.618749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:17.618405    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:05:26.669761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:26.669732    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-f9d557c9-9x2z8"
Nov 01 20:05:26.711370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:26.711339    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:05:26.711580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:26.711536    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" podUID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" containerName="manager" containerID="cri-o://d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7" gracePeriod=30
Nov 01 20:05:26.967063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:26.967038    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:05:27.005130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.005098    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\" (UniqueName: \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\") pod \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") "
Nov 01 20:05:27.005313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.005167    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\") pod \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\" (UID: \"e27d37b4-1d54-4b46-9d4f-50b0af39744d\") "
Nov 01 20:05:27.007245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.007214    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb" pod "e27d37b4-1d54-4b46-9d4f-50b0af39744d" (UID: "e27d37b4-1d54-4b46-9d4f-50b0af39744d"). InnerVolumeSpecName "kube-api-access-jsslb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:05:27.007359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.007219    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs" pod "e27d37b4-1d54-4b46-9d4f-50b0af39744d" (UID: "e27d37b4-1d54-4b46-9d4f-50b0af39744d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:05:27.106117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.106085    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsslb\" (UniqueName: \"kubernetes.io/projected/e27d37b4-1d54-4b46-9d4f-50b0af39744d-kube-api-access-jsslb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:05:27.106117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.106115    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e27d37b4-1d54-4b46-9d4f-50b0af39744d-metrics-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:05:27.723342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.723305    5041 generic.go:356] "Generic (PLEG): container finished" podID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" containerID="d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7" exitCode=0
Nov 01 20:05:27.723765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.723357    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq"
Nov 01 20:05:27.723765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.723387    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" event={"ID":"e27d37b4-1d54-4b46-9d4f-50b0af39744d","Type":"ContainerDied","Data":"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"}
Nov 01 20:05:27.723765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.723449    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-rgmfq" event={"ID":"e27d37b4-1d54-4b46-9d4f-50b0af39744d","Type":"ContainerDied","Data":"5b553f70e5f93afc9c22d8e499940a93aca0dc50d4a793b307c20f5c68c4f498"}
Nov 01 20:05:27.723765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.723471    5041 scope.go:122] "RemoveContainer" containerID="d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"
Nov 01 20:05:27.733153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.733135    5041 scope.go:122] "RemoveContainer" containerID="d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"
Nov 01 20:05:27.733432 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:05:27.733412    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7\": container with ID starting with d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7 not found: ID does not exist" containerID="d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"
Nov 01 20:05:27.733483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.733441    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7"} err="failed to get container status \"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7\": rpc error: code = NotFound desc = could not find container \"d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7\": container with ID starting with d6c4eae68c66a33429feea9d2a91d7a1ba9bfc02e990cae9d266f02b55678db7 not found: ID does not exist"
Nov 01 20:05:27.744576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.744553    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:05:27.748437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:27.748415    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-rgmfq"]
Nov 01 20:05:28.426554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:28.426514    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" path="/var/lib/kubelet/pods/e27d37b4-1d54-4b46-9d4f-50b0af39744d/volumes"
Nov 01 20:05:28.540149 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:05:28.540114    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:05:32.494763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.494731    5041 scope.go:122] "RemoveContainer" containerID="f8e6d3e7ea03c51c76fc2b7511908030e4cf8776eb315f2f97917b69817b4df2"
Nov 01 20:05:32.594490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.594455    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"]
Nov 01 20:05:32.594904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.594890    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" containerName="manager"
Nov 01 20:05:32.594963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.594905    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" containerName="manager"
Nov 01 20:05:32.595014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.595004    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e27d37b4-1d54-4b46-9d4f-50b0af39744d" containerName="manager"
Nov 01 20:05:32.599666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.599643    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.612068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.612041    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"]
Nov 01 20:05:32.612241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.612226    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ww25v\""
Nov 01 20:05:32.612284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.612274    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 20:05:32.612330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.612320    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 20:05:32.612929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.612894    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 20:05:32.652429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652397    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.652600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.652600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652536    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.652691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4fd9f8-3aea-42af-8483-acb9f8198223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.652736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmsc6\" (UniqueName: \"kubernetes.io/projected/dc4fd9f8-3aea-42af-8483-acb9f8198223-kube-api-access-cmsc6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.652736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.652727    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753310    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753346    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753369    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753398    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4fd9f8-3aea-42af-8483-acb9f8198223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753426    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmsc6\" (UniqueName: \"kubernetes.io/projected/dc4fd9f8-3aea-42af-8483-acb9f8198223-kube-api-access-cmsc6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753448    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753816    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753902    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.753986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.753913    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.755756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.755734    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc4fd9f8-3aea-42af-8483-acb9f8198223-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.755948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.755931    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc4fd9f8-3aea-42af-8483-acb9f8198223-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.761494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.761472    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmsc6\" (UniqueName: \"kubernetes.io/projected/dc4fd9f8-3aea-42af-8483-acb9f8198223-kube-api-access-cmsc6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2\" (UID: \"dc4fd9f8-3aea-42af-8483-acb9f8198223\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:32.916574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:32.916540    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:33.192491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:33.192459    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"]
Nov 01 20:05:33.195054 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:33.194999    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4fd9f8_3aea_42af_8483_acb9f8198223.slice/crio-c31a1ffb9f9edbf294eeb0bfa249133611c86c899f10c2f11d506edf013ca018 WatchSource:0}: Error finding container c31a1ffb9f9edbf294eeb0bfa249133611c86c899f10c2f11d506edf013ca018: Status 404 returned error can't find the container with id c31a1ffb9f9edbf294eeb0bfa249133611c86c899f10c2f11d506edf013ca018
Nov 01 20:05:33.764392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:33.764342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2" event={"ID":"dc4fd9f8-3aea-42af-8483-acb9f8198223","Type":"ContainerStarted","Data":"c31a1ffb9f9edbf294eeb0bfa249133611c86c899f10c2f11d506edf013ca018"}
Nov 01 20:05:34.771271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:34.771230    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2" event={"ID":"dc4fd9f8-3aea-42af-8483-acb9f8198223","Type":"ContainerStarted","Data":"6e3483bd8f2c56f85e07caaec9c97ca58e6922181e7ade5ecc22479a03002bcd"}
Nov 01 20:05:34.789438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:34.789394    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2" podStartSLOduration=1.307336019 podStartE2EDuration="2.789376494s" totalImagesPullingTime="1.482040475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:34.787164768 +0000 UTC m=+422.508243483" watchObservedRunningTime="2026-11-01 20:05:34.789376494 +0000 UTC m=+422.510455170"
Nov 01 20:05:34.917053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:34.917024    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:35.226744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.226710    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"]
Nov 01 20:05:35.230652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.230631    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.239582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.239561    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 01 20:05:35.241441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.241420    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"]
Nov 01 20:05:35.290538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290506    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.290705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290544    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67dsc\" (UniqueName: \"kubernetes.io/projected/e79a81c6-16a9-4659-b063-05f6285aa2db-kube-api-access-67dsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.290705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290577    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.290868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290699    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.290868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290853    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.290958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.290883    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79a81c6-16a9-4659-b063-05f6285aa2db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.391922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.391888    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.391955    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.391971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79a81c6-16a9-4659-b063-05f6285aa2db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.391993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.392017    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67dsc\" (UniqueName: \"kubernetes.io/projected/e79a81c6-16a9-4659-b063-05f6285aa2db-kube-api-access-67dsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.392048    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.392303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.392375    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.392447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.392422    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.394270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.394243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e79a81c6-16a9-4659-b063-05f6285aa2db-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.394667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.394650    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79a81c6-16a9-4659-b063-05f6285aa2db-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.399560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.399540    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67dsc\" (UniqueName: \"kubernetes.io/projected/e79a81c6-16a9-4659-b063-05f6285aa2db-kube-api-access-67dsc\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-qvfwp\" (UID: \"e79a81c6-16a9-4659-b063-05f6285aa2db\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.545711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.545681    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:35.820961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:35.820932    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"]
Nov 01 20:05:35.824487 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:35.824455    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79a81c6_16a9_4659_b063_05f6285aa2db.slice/crio-53db34180846f6fea2aa053acb86362eca7b7dbef7b603e2cffb4c8d4576a3f3 WatchSource:0}: Error finding container 53db34180846f6fea2aa053acb86362eca7b7dbef7b603e2cffb4c8d4576a3f3: Status 404 returned error can't find the container with id 53db34180846f6fea2aa053acb86362eca7b7dbef7b603e2cffb4c8d4576a3f3
Nov 01 20:05:36.782434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:36.782397    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp" event={"ID":"e79a81c6-16a9-4659-b063-05f6285aa2db","Type":"ContainerStarted","Data":"98814a5c7f1cea43f7dd00e05892630749eab8f135db2a051866d8647c6e670b"}
Nov 01 20:05:36.782632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:36.782454    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp" event={"ID":"e79a81c6-16a9-4659-b063-05f6285aa2db","Type":"ContainerStarted","Data":"53db34180846f6fea2aa053acb86362eca7b7dbef7b603e2cffb4c8d4576a3f3"}
Nov 01 20:05:36.802165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:36.802120    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp" podStartSLOduration=1.5122378250000001 podStartE2EDuration="1.802107935s" totalImagesPullingTime="289.87011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:36.798939329 +0000 UTC m=+424.520018010" watchObservedRunningTime="2026-11-01 20:05:36.802107935 +0000 UTC m=+424.523186611"
Nov 01 20:05:37.546225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:37.546186    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:38.467004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.466968    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"]
Nov 01 20:05:38.472790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.472748    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.479494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.479463    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"]
Nov 01 20:05:38.480138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.480036    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 20:05:38.520391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520361    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k28l\" (UniqueName: \"kubernetes.io/projected/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-kube-api-access-9k28l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.520559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520401    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.520559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.520658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520563    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.520658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520594    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.520658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.520652    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.621900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.621861    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.621920    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.621963    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.621986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.622041    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k28l\" (UniqueName: \"kubernetes.io/projected/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-kube-api-access-9k28l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.622076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.622388    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.622433    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.622605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.622475    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.624423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.624393    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.624589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.624570    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.630279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.630253    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k28l\" (UniqueName: \"kubernetes.io/projected/1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb-kube-api-access-9k28l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-94gsr\" (UID: \"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:38.791091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:38.791059    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:39.070534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:39.070454    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"]
Nov 01 20:05:39.075681 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:39.075621    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdc2c1c_0ee4_4b2e_83dc_29327158e7eb.slice/crio-2c107111f332138a9c58dceea22ee3708e917aec198d8e01c16194bf75d6d953 WatchSource:0}: Error finding container 2c107111f332138a9c58dceea22ee3708e917aec198d8e01c16194bf75d6d953: Status 404 returned error can't find the container with id 2c107111f332138a9c58dceea22ee3708e917aec198d8e01c16194bf75d6d953
Nov 01 20:05:39.799271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:39.799235    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr" event={"ID":"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb","Type":"ContainerStarted","Data":"c457699ad56e19a870eabe3230bfd68511ba8646a12b100d3d9be92f93689ceb"}
Nov 01 20:05:39.799753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:39.799299    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr" event={"ID":"1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb","Type":"ContainerStarted","Data":"2c107111f332138a9c58dceea22ee3708e917aec198d8e01c16194bf75d6d953"}
Nov 01 20:05:39.816749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:39.816691    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr" podStartSLOduration=1.509379695 podStartE2EDuration="1.816678314s" totalImagesPullingTime="307.298619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:39.815239567 +0000 UTC m=+427.536318255" watchObservedRunningTime="2026-11-01 20:05:39.816678314 +0000 UTC m=+427.537756989"
Nov 01 20:05:40.791706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:40.791671    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:41.063498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.063421    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"]
Nov 01 20:05:41.072475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.072424    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.076049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.076022    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"]
Nov 01 20:05:41.082897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.082856    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 20:05:41.150011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.149979    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.150186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.150020    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndbq\" (UniqueName: \"kubernetes.io/projected/1342869f-efd5-4140-bb37-c064736dfc96-kube-api-access-dndbq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.150186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.150096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.150186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.150127    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1342869f-efd5-4140-bb37-c064736dfc96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.150302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.150265    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.150342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.150328    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251449    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251506    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251527    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251545    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndbq\" (UniqueName: \"kubernetes.io/projected/1342869f-efd5-4140-bb37-c064736dfc96-kube-api-access-dndbq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251564    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251581    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1342869f-efd5-4140-bb37-c064736dfc96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251884    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.251983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.251944    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.252056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.252028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.253954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.253923    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1342869f-efd5-4140-bb37-c064736dfc96-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.254101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.254084    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1342869f-efd5-4140-bb37-c064736dfc96-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.259602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.259581    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndbq\" (UniqueName: \"kubernetes.io/projected/1342869f-efd5-4140-bb37-c064736dfc96-kube-api-access-dndbq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c\" (UID: \"1342869f-efd5-4140-bb37-c064736dfc96\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.403390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.403305    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:41.681074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.681045    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"]
Nov 01 20:05:41.683116 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:41.683089    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1342869f_efd5_4140_bb37_c064736dfc96.slice/crio-37d1b37c216d30efca5106427c3b0548678e2b7ca932894b6cafb7fa533791c8 WatchSource:0}: Error finding container 37d1b37c216d30efca5106427c3b0548678e2b7ca932894b6cafb7fa533791c8: Status 404 returned error can't find the container with id 37d1b37c216d30efca5106427c3b0548678e2b7ca932894b6cafb7fa533791c8
Nov 01 20:05:41.812032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:41.812001    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c" event={"ID":"1342869f-efd5-4140-bb37-c064736dfc96","Type":"ContainerStarted","Data":"37d1b37c216d30efca5106427c3b0548678e2b7ca932894b6cafb7fa533791c8"}
Nov 01 20:05:42.462330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.462296    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"]
Nov 01 20:05:42.466692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.466668    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.475050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.475024    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"]
Nov 01 20:05:42.479522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.478921    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 20:05:42.566284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566238    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.566455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566295    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.566455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566379    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx77l\" (UniqueName: \"kubernetes.io/projected/5a81bba4-497d-453e-844a-a9e5cbc1f94b-kube-api-access-cx77l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.566455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566442    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.566641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566476    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.566641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.566516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.667955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.667925    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.667992    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668139    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668166    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668191    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx77l\" (UniqueName: \"kubernetes.io/projected/5a81bba4-497d-453e-844a-a9e5cbc1f94b-kube-api-access-cx77l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668218    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668393    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668538    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.668693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.668676    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.671067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.671036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a81bba4-497d-453e-844a-a9e5cbc1f94b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.671213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.671133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a81bba4-497d-453e-844a-a9e5cbc1f94b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.676921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.676886    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx77l\" (UniqueName: \"kubernetes.io/projected/5a81bba4-497d-453e-844a-a9e5cbc1f94b-kube-api-access-cx77l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5\" (UID: \"5a81bba4-497d-453e-844a-a9e5cbc1f94b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.784962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.784925    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:42.820740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.820700    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c" event={"ID":"1342869f-efd5-4140-bb37-c064736dfc96","Type":"ContainerStarted","Data":"7f766988f457f17b92c37e4647e7398f1cb8ae62f946808710835e94336d809c"}
Nov 01 20:05:42.842145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.842038    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c" podStartSLOduration=1.5189665300000001 podStartE2EDuration="1.842007808s" totalImagesPullingTime="323.041278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:42.840645128 +0000 UTC m=+430.561723805" watchObservedRunningTime="2026-11-01 20:05:42.842007808 +0000 UTC m=+430.563086478"
Nov 01 20:05:42.917283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.917236    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:42.930112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:42.930086    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:43.071348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.071225    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"]
Nov 01 20:05:43.074583 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:43.074554    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a81bba4_497d_453e_844a_a9e5cbc1f94b.slice/crio-d136e27a93a882a159d792e06cf20494a44044d213012e79a9c4e69871481408 WatchSource:0}: Error finding container d136e27a93a882a159d792e06cf20494a44044d213012e79a9c4e69871481408: Status 404 returned error can't find the container with id d136e27a93a882a159d792e06cf20494a44044d213012e79a9c4e69871481408
Nov 01 20:05:43.403411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.403388    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:43.827351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.827318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" event={"ID":"5a81bba4-497d-453e-844a-a9e5cbc1f94b","Type":"ContainerStarted","Data":"08689a2e6d63a883082894fe8ad4e1a251c1f418d9e49f7a68c843922a9c26e0"}
Nov 01 20:05:43.827817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.827363    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" event={"ID":"5a81bba4-497d-453e-844a-a9e5cbc1f94b","Type":"ContainerStarted","Data":"d136e27a93a882a159d792e06cf20494a44044d213012e79a9c4e69871481408"}
Nov 01 20:05:43.838646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.838616    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2"
Nov 01 20:05:43.844561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:43.844519    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podStartSLOduration=1.541824703 podStartE2EDuration="1.844507193s" totalImagesPullingTime="302.68249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:43.84354545 +0000 UTC m=+431.564624126" watchObservedRunningTime="2026-11-01 20:05:43.844507193 +0000 UTC m=+431.565585869"
Nov 01 20:05:44.785145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:44.785113    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:45.546890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:45.546844    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:45.564734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:45.564660    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:45.852131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:45.852048    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-qvfwp"
Nov 01 20:05:48.791999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:48.791960    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:48.804998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:48.804971    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:48.870200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:48.870165    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-94gsr"
Nov 01 20:05:51.404534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:51.404496    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:51.422541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:51.422512    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:51.886592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:51.886566    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c"
Nov 01 20:05:52.785445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:52.785405    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:52.798241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:52.798210    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:52.892642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:52.892611    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:05:55.131485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.131444    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d8bc659cc-72rg2"]
Nov 01 20:05:55.135250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.135233    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.142142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.142114    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d8bc659cc-72rg2"]
Nov 01 20:05:55.295216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.295172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a08d27-42f8-48e2-8511-0032f24c6fd6-tls-cert\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.295368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.295305    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bl7x\" (UniqueName: \"kubernetes.io/projected/40a08d27-42f8-48e2-8511-0032f24c6fd6-kube-api-access-7bl7x\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.396849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.396728    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bl7x\" (UniqueName: \"kubernetes.io/projected/40a08d27-42f8-48e2-8511-0032f24c6fd6-kube-api-access-7bl7x\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.397001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.396875    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a08d27-42f8-48e2-8511-0032f24c6fd6-tls-cert\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.399422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.399392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/40a08d27-42f8-48e2-8511-0032f24c6fd6-tls-cert\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.404343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.404319    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bl7x\" (UniqueName: \"kubernetes.io/projected/40a08d27-42f8-48e2-8511-0032f24c6fd6-kube-api-access-7bl7x\") pod \"authorino-6d8bc659cc-72rg2\" (UID: \"40a08d27-42f8-48e2-8511-0032f24c6fd6\") " pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.449116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.449080    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8bc659cc-72rg2"
Nov 01 20:05:55.721298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.721272    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d8bc659cc-72rg2"]
Nov 01 20:05:55.723260 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:05:55.723234    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40a08d27_42f8_48e2_8511_0032f24c6fd6.slice/crio-d2318959c567f28bd35ed38c2ccb24f7c7effa0a4e932602b319bb3a4fe998f5 WatchSource:0}: Error finding container d2318959c567f28bd35ed38c2ccb24f7c7effa0a4e932602b319bb3a4fe998f5: Status 404 returned error can't find the container with id d2318959c567f28bd35ed38c2ccb24f7c7effa0a4e932602b319bb3a4fe998f5
Nov 01 20:05:55.898944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:55.898913    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8bc659cc-72rg2" event={"ID":"40a08d27-42f8-48e2-8511-0032f24c6fd6","Type":"ContainerStarted","Data":"d2318959c567f28bd35ed38c2ccb24f7c7effa0a4e932602b319bb3a4fe998f5"}
Nov 01 20:05:56.906823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:56.906759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d8bc659cc-72rg2" event={"ID":"40a08d27-42f8-48e2-8511-0032f24c6fd6","Type":"ContainerStarted","Data":"c88aeaf7e842e9c4c577f4b7545cc008456e2b0841ddf949202d6073bbcf58d7"}
Nov 01 20:05:56.922190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:56.922144    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d8bc659cc-72rg2" podStartSLOduration=0.926913715 podStartE2EDuration="1.922131815s" totalImagesPullingTime="995.2181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:05:56.920989413 +0000 UTC m=+444.642068090" watchObservedRunningTime="2026-11-01 20:05:56.922131815 +0000 UTC m=+444.643210491"
Nov 01 20:05:56.946285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:56.946250    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:05:56.946718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:56.946691    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-564698f58b-mv8bw" podUID="dabfeb27-eecb-4658-a6fc-cb32a53300da" containerName="authorino" containerID="cri-o://4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa" gracePeriod=30
Nov 01 20:05:57.214965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.214938    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:05:57.317985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.317951    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\" (UniqueName: \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\") pod \"dabfeb27-eecb-4658-a6fc-cb32a53300da\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") "
Nov 01 20:05:57.318176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.318113    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\" (UniqueName: \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\") pod \"dabfeb27-eecb-4658-a6fc-cb32a53300da\" (UID: \"dabfeb27-eecb-4658-a6fc-cb32a53300da\") "
Nov 01 20:05:57.320135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.320104    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx" pod "dabfeb27-eecb-4658-a6fc-cb32a53300da" (UID: "dabfeb27-eecb-4658-a6fc-cb32a53300da"). InnerVolumeSpecName "kube-api-access-r67hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:05:57.330126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.330094    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert" pod "dabfeb27-eecb-4658-a6fc-cb32a53300da" (UID: "dabfeb27-eecb-4658-a6fc-cb32a53300da"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:05:57.419728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.419692    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67hx\" (UniqueName: \"kubernetes.io/projected/dabfeb27-eecb-4658-a6fc-cb32a53300da-kube-api-access-r67hx\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:05:57.419728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.419722    5041 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dabfeb27-eecb-4658-a6fc-cb32a53300da-tls-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:05:57.917954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.917918    5041 generic.go:356] "Generic (PLEG): container finished" podID="dabfeb27-eecb-4658-a6fc-cb32a53300da" containerID="4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa" exitCode=0
Nov 01 20:05:57.918446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.917982    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-564698f58b-mv8bw"
Nov 01 20:05:57.918446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.918006    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564698f58b-mv8bw" event={"ID":"dabfeb27-eecb-4658-a6fc-cb32a53300da","Type":"ContainerDied","Data":"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"}
Nov 01 20:05:57.918446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.918060    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-564698f58b-mv8bw" event={"ID":"dabfeb27-eecb-4658-a6fc-cb32a53300da","Type":"ContainerDied","Data":"1fd29aa891687f5243b94e8b5e2c3fd59a6fe0607e9471c2021cf460075f18a7"}
Nov 01 20:05:57.918446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.918085    5041 scope.go:122] "RemoveContainer" containerID="4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"
Nov 01 20:05:57.928853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.928835    5041 scope.go:122] "RemoveContainer" containerID="4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"
Nov 01 20:05:57.929196 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:05:57.929175    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa\": container with ID starting with 4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa not found: ID does not exist" containerID="4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"
Nov 01 20:05:57.929315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.929201    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa"} err="failed to get container status \"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa\": rpc error: code = NotFound desc = could not find container \"4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa\": container with ID starting with 4a2a6aa7dbebc66d2bcb2f0a8a8709b00ecabdbb24970ed9c8c015222245d6aa not found: ID does not exist"
Nov 01 20:05:57.948432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.948407    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:05:57.956285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:57.956262    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-564698f58b-mv8bw"]
Nov 01 20:05:58.429663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:05:58.429599    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabfeb27-eecb-4658-a6fc-cb32a53300da" path="/var/lib/kubelet/pods/dabfeb27-eecb-4658-a6fc-cb32a53300da/volumes"
Nov 01 20:05:58.541752 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:05:58.541717    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:06:07.689722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.689684    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:06:07.690220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.690172    5041 cpu_manager.go:414] "containerMap: removing container" podUID="dabfeb27-eecb-4658-a6fc-cb32a53300da" containerName="authorino"
Nov 01 20:06:07.690220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.690184    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabfeb27-eecb-4658-a6fc-cb32a53300da" containerName="authorino"
Nov 01 20:06:07.690292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.690266    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabfeb27-eecb-4658-a6fc-cb32a53300da" containerName="authorino"
Nov 01 20:06:07.696730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.696709    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.706428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.706403    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-njf76\""
Nov 01 20:06:07.709878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.709854    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:06:07.711751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.711725    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:06:07.716289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.716267    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.724713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724682    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.724867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724731    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.724867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.724867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724825    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724913    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwvs\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724941    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724969    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.724995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.725061    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.725620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.725596    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:06:07.826321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826287    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826335    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826445    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826484    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826507    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826599    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826630    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826708    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cscl\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.826965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwvs\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826899    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.826965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826964    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.826994    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.827134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827020    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827072    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827127    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.827460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827071    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.827460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827259    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827280    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.827660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.827519    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.829248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.829227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.829349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.829311    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.834676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.834656    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.834910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.834887    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwvs\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\") pod \"maas-default-gateway-openshift-default-fd4b99879-8v5dc\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:07.928229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928193    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928236    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928256    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928282    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928312    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cscl\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928348    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928395    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928418    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928485    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928793    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928815    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.928946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928926    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.929121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.928982    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.929179 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.929137    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.931041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.931008    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.931194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.931177    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.936896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.936872    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cscl\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:07.937075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:07.937058    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6sln\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:08.010952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.010869    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:08.034740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.034714    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:08.291823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.291801    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:06:08.293471 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:08.293440    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b7612a_9446_4561_b1a5_059617ee2ff1.slice/crio-91f3e56aab6b3cd3a815cbddb183dd2984ff338eac64e24cb2a30f80097ad767 WatchSource:0}: Error finding container 91f3e56aab6b3cd3a815cbddb183dd2984ff338eac64e24cb2a30f80097ad767: Status 404 returned error can't find the container with id 91f3e56aab6b3cd3a815cbddb183dd2984ff338eac64e24cb2a30f80097ad767
Nov 01 20:06:08.295599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.295568    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.295668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.295628    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.295668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.295662    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.320340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.320319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:06:08.326821 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:08.326797    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10ac272_5665_4fdd_bc15_00763c2c49ca.slice/crio-a072718928a950194f332da55b8769b2f320caff26e07c7f61fe335bc9bb90b1 WatchSource:0}: Error finding container a072718928a950194f332da55b8769b2f320caff26e07c7f61fe335bc9bb90b1: Status 404 returned error can't find the container with id a072718928a950194f332da55b8769b2f320caff26e07c7f61fe335bc9bb90b1
Nov 01 20:06:08.328982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.328951    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.329053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.329015    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.329053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.329040    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:08.994807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.994070    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" event={"ID":"a10ac272-5665-4fdd-bc15-00763c2c49ca","Type":"ContainerStarted","Data":"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"}
Nov 01 20:06:08.994807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.994143    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" event={"ID":"a10ac272-5665-4fdd-bc15-00763c2c49ca","Type":"ContainerStarted","Data":"a072718928a950194f332da55b8769b2f320caff26e07c7f61fe335bc9bb90b1"}
Nov 01 20:06:08.996169 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.996144    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" event={"ID":"70b7612a-9446-4561-b1a5-059617ee2ff1","Type":"ContainerStarted","Data":"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"}
Nov 01 20:06:08.996333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:08.996317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" event={"ID":"70b7612a-9446-4561-b1a5-059617ee2ff1","Type":"ContainerStarted","Data":"91f3e56aab6b3cd3a815cbddb183dd2984ff338eac64e24cb2a30f80097ad767"}
Nov 01 20:06:09.011882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.011857    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:09.013846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.013818    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:09.013964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.013884    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:09.014539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.014502    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podStartSLOduration=2.014489319 podStartE2EDuration="2.014489319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:09.012646898 +0000 UTC m=+456.733725573" watchObservedRunningTime="2026-11-01 20:06:09.014489319 +0000 UTC m=+456.735567994"
Nov 01 20:06:09.033819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.033759    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podStartSLOduration=2.033746876 podStartE2EDuration="2.033746876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:09.031142515 +0000 UTC m=+456.752221191" watchObservedRunningTime="2026-11-01 20:06:09.033746876 +0000 UTC m=+456.754825551"
Nov 01 20:06:09.034991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.034973    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:09.036087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.036068    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:09.036169 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:09.036104    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:10.012224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:10.012195    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:10.012628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:10.012247    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:10.035827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:10.035773    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:10.035983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:10.035844    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:11.012145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:11.012108    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:11.012322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:11.012163    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:11.036006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:11.035971    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:11.036183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:11.036022    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:12.011314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:12.011280    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:12.011624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:12.011579    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:12.035604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:12.035327    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:12.035604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:12.035371    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:13.011565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:13.011524    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:13.011729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:13.011582    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:13.035406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:13.035378    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:13.035539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:13.035422    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:14.011509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:14.011470    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:14.011974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:14.011527    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:14.035208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:14.035173    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:14.035373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:14.035227    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:15.011505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:15.011461    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-8v5dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 01 20:06:15.011710 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:15.011530    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 01 20:06:15.035757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:15.035725    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6sln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Nov 01 20:06:15.035929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:15.035792    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Nov 01 20:06:16.588851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.588814    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:16.590044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.590013    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:16.591029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.590948    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:06:16.664160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.664113    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:16.665450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.665432    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:16.666240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:16.666217    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:06:22.745108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.745070    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:06:22.749344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.749324    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.765183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.765157    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:06:22.824071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824037    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824069    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824094    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824149    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824234    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824281    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824366    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96l5g\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824425    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.824546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.824530    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.925881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.925848    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.925894    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96l5g\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.925936    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.925978    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926043    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926068    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926109    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926154    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926200    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926515    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926567    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926818    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.926917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.926893    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.927165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.927144    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.928639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.928612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.928732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.928634    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.933684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.933655    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96l5g\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:22.933761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:22.933663    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-jmn77\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:23.063349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:23.063303    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:23.340080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:23.340040    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:06:23.341397 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:23.341372    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542477d7_d721_481b_8972_6339c6c82be2.slice/crio-d9308d3758ca26413faa568403cd20dd244e24b376c661e3ffd66f719e737779 WatchSource:0}: Error finding container d9308d3758ca26413faa568403cd20dd244e24b376c661e3ffd66f719e737779: Status 404 returned error can't find the container with id d9308d3758ca26413faa568403cd20dd244e24b376c661e3ffd66f719e737779
Nov 01 20:06:23.343459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:23.343424    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:23.343567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:23.343507    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:23.343629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:23.343577    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:24.087278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:24.087241    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" event={"ID":"542477d7-d721-481b-8972-6339c6c82be2","Type":"ContainerStarted","Data":"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"}
Nov 01 20:06:24.087664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:24.087298    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" event={"ID":"542477d7-d721-481b-8972-6339c6c82be2","Type":"ContainerStarted","Data":"d9308d3758ca26413faa568403cd20dd244e24b376c661e3ffd66f719e737779"}
Nov 01 20:06:24.108446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:24.108402    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podStartSLOduration=2.108388683 podStartE2EDuration="2.108388683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:24.105369376 +0000 UTC m=+471.826448053" watchObservedRunningTime="2026-11-01 20:06:24.108388683 +0000 UTC m=+471.829467358"
Nov 01 20:06:25.066134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.066099    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:25.068464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.068433    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:25.068609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.068497    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:25.894242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.892022    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:06:25.912709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.912664    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:25.914529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.914497    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:06:25.917954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:25.917929    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3a6480-openshift-default-dockercfg-7wlx7\""
Nov 01 20:06:26.067804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.067543    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:26.067804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.067673    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:26.087324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087283    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087327    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087360    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087391    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087428    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2p2r\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087459    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087476    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.087657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.087551    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188069    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188125    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188173    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188208    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188258    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188287    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188326    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2p2r\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188424    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188483    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.188835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.188747    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.189059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.189034    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.189211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.189109    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.189275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.189218    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.190832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.190811    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.190962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.190910    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.195906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.195871    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.196236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.196215    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2p2r\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.230101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.230080    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:26.606926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.606897    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:06:26.608000 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:26.607973    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3405bab_1ca0_433e_8c93_e3f0ad7e8b6d.slice/crio-846e5fd914130d5bd43fe0cf6208c20d0c5bd95835962f03d328fde16dfd8927 WatchSource:0}: Error finding container 846e5fd914130d5bd43fe0cf6208c20d0c5bd95835962f03d328fde16dfd8927: Status 404 returned error can't find the container with id 846e5fd914130d5bd43fe0cf6208c20d0c5bd95835962f03d328fde16dfd8927
Nov 01 20:06:26.610467 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.610430    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:26.610591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.610520    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:26.610591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:26.610572    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:27.067148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.067114    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:27.067707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.067675    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:27.105816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.105766    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerStarted","Data":"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"}
Nov 01 20:06:27.105976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.105839    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerStarted","Data":"846e5fd914130d5bd43fe0cf6208c20d0c5bd95835962f03d328fde16dfd8927"}
Nov 01 20:06:27.126906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.126858    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podStartSLOduration=2.1268452 podStartE2EDuration="2.1268452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:27.124926555 +0000 UTC m=+474.846005234" watchObservedRunningTime="2026-11-01 20:06:27.1268452 +0000 UTC m=+474.847923931"
Nov 01 20:06:27.231258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.231220    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:27.236676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:27.236648    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:28.064801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:28.064183    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:28.064801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:28.064246    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:28.111705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:28.111670    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:28.112712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:28.112693    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:28.553582 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:06:28.553497    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:06:29.064192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:29.064152    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:29.064474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:29.064445    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:30.064186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:30.064148    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-jmn77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Nov 01 20:06:30.064630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:30.064224    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Nov 01 20:06:30.964390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:30.964348    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:06:30.969345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:30.969324    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:30.991014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:30.990983    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:06:31.006156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.006128    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3a6480\""
Nov 01 20:06:31.019428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.019403    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:06:31.023341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.023319    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.040221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.040193    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3a6480-dockercfg-fxg8f\""
Nov 01 20:06:31.044969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.044946    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:06:31.146393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.146357    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxfj\" (UniqueName: \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.146851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.146442    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmlz\" (UniqueName: \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.146851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.146557    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.146851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.146746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.248056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.247958    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxfj\" (UniqueName: \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.248056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.248011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmlz\" (UniqueName: \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.248056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.248051    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.248353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.248159    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.248906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.248884    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.249176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.249148    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.256791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.256678    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmlz\" (UniqueName: \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\") pod \"payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.256989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.256963    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxfj\" (UniqueName: \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\") pod \"payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.284070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.284028    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:31.344506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.343865    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:31.570763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.570736    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:06:31.572222 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:31.572192    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65370231_0b33_48e1_8c1c_e8b23c347555.slice/crio-7d3e0751bdd44668bb445d21a151db5fe2c948d5fdfe6a56244bb762f5141d31 WatchSource:0}: Error finding container 7d3e0751bdd44668bb445d21a151db5fe2c948d5fdfe6a56244bb762f5141d31: Status 404 returned error can't find the container with id 7d3e0751bdd44668bb445d21a151db5fe2c948d5fdfe6a56244bb762f5141d31
Nov 01 20:06:31.669348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.669317    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:06:31.679890 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:31.679858    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e334ce_7226_460d_a52f_ac96ebe93f6c.slice/crio-6e468ac187bfb5d6370cfbbf0e7f68cf1593493b131414aee3c51f7347f13b20 WatchSource:0}: Error finding container 6e468ac187bfb5d6370cfbbf0e7f68cf1593493b131414aee3c51f7347f13b20: Status 404 returned error can't find the container with id 6e468ac187bfb5d6370cfbbf0e7f68cf1593493b131414aee3c51f7347f13b20
Nov 01 20:06:31.931600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.931520    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:31.931997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.931976    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:31.932903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:31.932882    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:06:32.144447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.144411    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" event={"ID":"65370231-0b33-48e1-8c1c-e8b23c347555","Type":"ContainerStarted","Data":"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"}
Nov 01 20:06:32.144639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.144478    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" event={"ID":"65370231-0b33-48e1-8c1c-e8b23c347555","Type":"ContainerStarted","Data":"7d3e0751bdd44668bb445d21a151db5fe2c948d5fdfe6a56244bb762f5141d31"}
Nov 01 20:06:32.144639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.144510    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:32.146134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.146106    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" event={"ID":"98e334ce-7226-460d-a52f-ac96ebe93f6c","Type":"ContainerStarted","Data":"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"}
Nov 01 20:06:32.146255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.146153    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" event={"ID":"98e334ce-7226-460d-a52f-ac96ebe93f6c","Type":"ContainerStarted","Data":"6e468ac187bfb5d6370cfbbf0e7f68cf1593493b131414aee3c51f7347f13b20"}
Nov 01 20:06:32.146321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.146295    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:32.166996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.166954    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" podStartSLOduration=2.166942209 podStartE2EDuration="2.166942209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:32.164361358 +0000 UTC m=+479.885440033" watchObservedRunningTime="2026-11-01 20:06:32.166942209 +0000 UTC m=+479.888020885"
Nov 01 20:06:32.186048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.185959    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" podStartSLOduration=2.185943973 podStartE2EDuration="2.185943973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:32.185670731 +0000 UTC m=+479.906749407" watchObservedRunningTime="2026-11-01 20:06:32.185943973 +0000 UTC m=+479.907022649"
Nov 01 20:06:32.386761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.386727    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:06:32.390672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.390655    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.401261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.401236    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:06:32.467347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.467254    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0aa61b\""
Nov 01 20:06:32.467515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.467402    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0aa61b-dockercfg-5grfn\""
Nov 01 20:06:32.567569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.567538    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.567722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.567591    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmbm\" (UniqueName: \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.668235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.668202    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.668429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.668260    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmbm\" (UniqueName: \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.669286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.669255    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.676658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.676615    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmbm\" (UniqueName: \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\") pod \"payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.708133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.708103    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:32.985566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:32.985532    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:06:32.987129 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:32.987098    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6f29ac_fbad_406f_8a3f_32a8ff40d427.slice/crio-113c51b7593bb611a7e3d0d8f207d4e233fd2be70f717ea38977dd1ae66752c5 WatchSource:0}: Error finding container 113c51b7593bb611a7e3d0d8f207d4e233fd2be70f717ea38977dd1ae66752c5: Status 404 returned error can't find the container with id 113c51b7593bb611a7e3d0d8f207d4e233fd2be70f717ea38977dd1ae66752c5
Nov 01 20:06:33.155373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:33.155323    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" event={"ID":"2f6f29ac-fbad-406f-8a3f-32a8ff40d427","Type":"ContainerStarted","Data":"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"}
Nov 01 20:06:33.155529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:33.155394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" event={"ID":"2f6f29ac-fbad-406f-8a3f-32a8ff40d427","Type":"ContainerStarted","Data":"113c51b7593bb611a7e3d0d8f207d4e233fd2be70f717ea38977dd1ae66752c5"}
Nov 01 20:06:33.156166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:33.156143    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:33.173791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:33.173738    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" podStartSLOduration=1.173725492 podStartE2EDuration="1.173725492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:33.171047447 +0000 UTC m=+480.892126124" watchObservedRunningTime="2026-11-01 20:06:33.173725492 +0000 UTC m=+480.894804232"
Nov 01 20:06:37.774830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.765947    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"]
Nov 01 20:06:37.784309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.784271    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"]
Nov 01 20:06:37.784445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.784390    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826271    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826379    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826449    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826501    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826570    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1388a62-2172-466e-91f6-dcef2cf7525b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826594    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztrb\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-kube-api-access-4ztrb\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826645    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.826701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.826666    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.927836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.927792    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.927895    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.927949    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.927990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928050    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928103    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928163    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1388a62-2172-466e-91f6-dcef2cf7525b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928197    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztrb\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-kube-api-access-4ztrb\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928433    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928565    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928693    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.928992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.928966    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1388a62-2172-466e-91f6-dcef2cf7525b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.930517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.930490    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.930687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.930667    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.939016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.938987    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:37.939286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:37.939257    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztrb\" (UniqueName: \"kubernetes.io/projected/b1388a62-2172-466e-91f6-dcef2cf7525b-kube-api-access-4ztrb\") pod \"maas-default-gateway-openshift-default-fd4b99879-bnsdt\" (UID: \"b1388a62-2172-466e-91f6-dcef2cf7525b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:38.099079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:38.099043    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:38.405083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:38.405053    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"]
Nov 01 20:06:38.406339 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:38.406311    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1388a62_2172_466e_91f6_dcef2cf7525b.slice/crio-0ccfb768f9d311e88bcb5e775031b52ae7448e449d4d9506a1af6ea5db3101d9 WatchSource:0}: Error finding container 0ccfb768f9d311e88bcb5e775031b52ae7448e449d4d9506a1af6ea5db3101d9: Status 404 returned error can't find the container with id 0ccfb768f9d311e88bcb5e775031b52ae7448e449d4d9506a1af6ea5db3101d9
Nov 01 20:06:38.408867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:38.408834    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:38.408944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:38.408902    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:38.408944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:38.408933    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:39.202270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:39.202232    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" event={"ID":"b1388a62-2172-466e-91f6-dcef2cf7525b","Type":"ContainerStarted","Data":"1e51eb866a6eda6b751e4743cb9b3301fc350eeda3e007118dc399e6c96052d2"}
Nov 01 20:06:39.202925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:39.202290    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" event={"ID":"b1388a62-2172-466e-91f6-dcef2cf7525b","Type":"ContainerStarted","Data":"0ccfb768f9d311e88bcb5e775031b52ae7448e449d4d9506a1af6ea5db3101d9"}
Nov 01 20:06:39.223333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:39.223287    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podStartSLOduration=2.223274194 podStartE2EDuration="2.223274194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:39.22059051 +0000 UTC m=+486.941669187" watchObservedRunningTime="2026-11-01 20:06:39.223274194 +0000 UTC m=+486.944352870"
Nov 01 20:06:40.099835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:40.099762    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:40.101482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:40.101453    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:40.101602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:40.101499    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:41.099523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:41.099481    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:41.099967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:41.099564    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:42.099726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:42.099689    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:42.100191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:42.099861    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:43.099846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:43.099715    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:43.099846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:43.099819    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:43.168938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:43.168898    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:06:43.169750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:43.169727    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:06:44.099539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:44.099504    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:44.099714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:44.099557    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:44.162269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:44.161905    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:06:44.162269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:44.161973    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:06:45.100219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:45.100184    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:45.100387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:45.100255    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:45.171683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:45.171651    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:06:46.099748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.099703    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Nov 01 20:06:46.100018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.099758    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Nov 01 20:06:46.254313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.254260    5041 generic.go:356] "Generic (PLEG): container finished" podID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerID="ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e" exitCode=0
Nov 01 20:06:46.254313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.254299    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerDied","Data":"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"}
Nov 01 20:06:46.254928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.254798    5041 scope.go:122] "RemoveContainer" containerID="ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"
Nov 01 20:06:46.256286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.256251    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:46.256438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.256319    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:46.256438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:46.256356    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:06:47.274013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.273973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerStarted","Data":"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"}
Nov 01 20:06:47.298510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.298480    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:47.298818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.298759    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:47.299671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.299651    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt"
Nov 01 20:06:47.787075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.787040    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"]
Nov 01 20:06:47.792248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.792218    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.810888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.810854    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:06:47.811524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.811494    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"]
Nov 01 20:06:47.814530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.814508    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:06:47.814644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.814538    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:06:47.814644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.814575    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:06:47.888959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.888912    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.889141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.888961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cs7w\" (UniqueName: \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.889141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.889020    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.990064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.990027    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.990269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.990162    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.990269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.990185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cs7w\" (UniqueName: \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.990750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.990721    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.992605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.992583    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:47.998524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:47.998503    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cs7w\" (UniqueName: \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\") pod \"maas-api-revoke-sub-af23ea43-s7blm\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:48.105332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:48.105232    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:48.231538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:48.231498    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:48.231696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:48.231645    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:06:48.231806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:48.231694    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:06:48.497180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:48.494622    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"]
Nov 01 20:06:48.500049 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:06:48.500011    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd996cd75_5930_4285_9803_5a315b1c8097.slice/crio-fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1 WatchSource:0}: Error finding container fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1: Status 404 returned error can't find the container with id fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1
Nov 01 20:06:49.230933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:49.230887    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:06:49.231108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:49.230974    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:06:49.365804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:49.357487    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm" event={"ID":"d996cd75-5930-4285-9803-5a315b1c8097","Type":"ContainerStarted","Data":"fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1"}
Nov 01 20:06:50.252811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:50.243165    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:06:50.252811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:50.243227    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:06:51.282424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:51.282385    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:06:51.283081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:51.283040    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:06:52.231418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:52.231378    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Nov 01 20:06:52.231613 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:52.231442    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Nov 01 20:06:53.692249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:53.692218    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:53.695874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:53.693727    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:53.695874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:53.695060    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:06:55.550175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:55.550139    5041 generic.go:356] "Generic (PLEG): container finished" podID="d996cd75-5930-4285-9803-5a315b1c8097" containerID="ef670f2c39a98df7f3a53668e36a624fb61695a287a0b8df6a8efad5441ccfa9" exitCode=0
Nov 01 20:06:55.551764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:55.551736    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm" event={"ID":"d996cd75-5930-4285-9803-5a315b1c8097","Type":"ContainerDied","Data":"ef670f2c39a98df7f3a53668e36a624fb61695a287a0b8df6a8efad5441ccfa9"}
Nov 01 20:06:57.403803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.403520    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:57.501476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.501442    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\") pod \"d996cd75-5930-4285-9803-5a315b1c8097\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") "
Nov 01 20:06:57.501844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.501812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\" (UniqueName: \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\") pod \"d996cd75-5930-4285-9803-5a315b1c8097\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") "
Nov 01 20:06:57.502150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.502130    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\") pod \"d996cd75-5930-4285-9803-5a315b1c8097\" (UID: \"d996cd75-5930-4285-9803-5a315b1c8097\") "
Nov 01 20:06:57.503411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.503385    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca" pod "d996cd75-5930-4285-9803-5a315b1c8097" (UID: "d996cd75-5930-4285-9803-5a315b1c8097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:06:57.506852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.506748    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca" pod "d996cd75-5930-4285-9803-5a315b1c8097" (UID: "d996cd75-5930-4285-9803-5a315b1c8097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:06:57.508481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.508444    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w" pod "d996cd75-5930-4285-9803-5a315b1c8097" (UID: "d996cd75-5930-4285-9803-5a315b1c8097"). InnerVolumeSpecName "kube-api-access-2cs7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:06:57.579226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.579191    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"
Nov 01 20:06:57.579398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.579224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm" event={"ID":"d996cd75-5930-4285-9803-5a315b1c8097","Type":"ContainerDied","Data":"fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1"}
Nov 01 20:06:57.579398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.579284    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbccada6c09056e8d69115651dc6b9bc075e6072320dfc50c60b03187a9089b1"
Nov 01 20:06:57.604406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.604378    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:06:57.604406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.604406    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cs7w\" (UniqueName: \"kubernetes.io/projected/d996cd75-5930-4285-9803-5a315b1c8097-kube-api-access-2cs7w\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:06:57.604578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:06:57.604421    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996cd75-5930-4285-9803-5a315b1c8097-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:06:58.540171 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:06:58.540138    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:07:02.489836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.489766    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:07:02.490436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.490418    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d996cd75-5930-4285-9803-5a315b1c8097" containerName="revoke-keys"
Nov 01 20:07:02.490501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.490440    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d996cd75-5930-4285-9803-5a315b1c8097" containerName="revoke-keys"
Nov 01 20:07:02.490574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.490563    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d996cd75-5930-4285-9803-5a315b1c8097" containerName="revoke-keys"
Nov 01 20:07:02.495592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.495573    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.502142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.501926    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"default-dockercfg-98hrf\""
Nov 01 20:07:02.502142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.501937    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"premium-w0-3a6480-kserve-self-signed-certs\""
Nov 01 20:07:02.502142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.501958    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:07:02.529500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.529463    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"openshift-service-ca.crt\""
Nov 01 20:07:02.529656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.529578    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"kube-root-ca.crt\""
Nov 01 20:07:02.658578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658539    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cm7s\" (UniqueName: \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.658786 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658620    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.658786 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658693    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.658786 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658723    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.658922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658802    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.658922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.658911    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.760571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760477    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cm7s\" (UniqueName: \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.760571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760567    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.760840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760592    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.760840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760616    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.760840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760758    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.761087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.760904    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.761171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.761125    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.761346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.761326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.761416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.761389    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.763215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.763193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.763499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.763482    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.771186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.771159    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cm7s\" (UniqueName: \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\") pod \"premium-w0-3a6480-kserve-67c95c7964-z9smt\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") " pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:02.811325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:02.811285    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:03.095095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:03.095069    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:07:03.096194 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:03.096169    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc63cfe70_ac77_474b_a224_5d2450af17b4.slice/crio-50768f96cbdaf8c934dc182ce5780497a171306e154bb23937d3df5ae2be1b71 WatchSource:0}: Error finding container 50768f96cbdaf8c934dc182ce5780497a171306e154bb23937d3df5ae2be1b71: Status 404 returned error can't find the container with id 50768f96cbdaf8c934dc182ce5780497a171306e154bb23937d3df5ae2be1b71
Nov 01 20:07:03.619629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:03.619590    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" event={"ID":"c63cfe70-ac77-474b-a224-5d2450af17b4","Type":"ContainerStarted","Data":"537078708ca0a6af3f0252e0c67c4608c462caa54d6bd5c797b4b3859bfde31a"}
Nov 01 20:07:03.619629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:03.619635    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" event={"ID":"c63cfe70-ac77-474b-a224-5d2450af17b4","Type":"ContainerStarted","Data":"50768f96cbdaf8c934dc182ce5780497a171306e154bb23937d3df5ae2be1b71"}
Nov 01 20:07:03.638575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:03.638523    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podStartSLOduration=1.6385081970000002 podStartE2EDuration="1.638508197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:03.634510699 +0000 UTC m=+511.355589375" watchObservedRunningTime="2026-11-01 20:07:03.638508197 +0000 UTC m=+511.359586872"
Nov 01 20:07:03.811541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:03.811507    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:07.918494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.918457    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:07:07.923980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.923955    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:07.930930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.930903    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:07:07.935003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.934977    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"premium-w2-e27ae7-kserve-self-signed-certs\""
Nov 01 20:07:07.935199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.935185    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"openshift-service-ca.crt\""
Nov 01 20:07:07.935383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.935368    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"default-dockercfg-cqft8\""
Nov 01 20:07:07.935493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:07.933476    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"kube-root-ca.crt\""
Nov 01 20:07:08.020714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.020680    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.020932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.020725    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.020932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.020749    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.021057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.020951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsggs\" (UniqueName: \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.021102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.021053    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.021156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.021100    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122020    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122089    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122166    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122269    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122392    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsggs\" (UniqueName: \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122452    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122517    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.122607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.122574    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.124364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.124339    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.124627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.124610    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.130027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.130002    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsggs\" (UniqueName: \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\") pod \"premium-w2-e27ae7-kserve-6dd5755d97-7wpsd\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") " pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.242038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.241959    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:08.519856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.519826    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:07:08.520949 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:08.520920    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919c5973_9291_4d84_bdda_5f203248be84.slice/crio-93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea WatchSource:0}: Error finding container 93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea: Status 404 returned error can't find the container with id 93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea
Nov 01 20:07:08.652722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.652642    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" event={"ID":"919c5973-9291-4d84-bdda-5f203248be84","Type":"ContainerStarted","Data":"a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb"}
Nov 01 20:07:08.652722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.652696    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" event={"ID":"919c5973-9291-4d84-bdda-5f203248be84","Type":"ContainerStarted","Data":"93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea"}
Nov 01 20:07:08.670824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:08.670751    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podStartSLOduration=1.670737345 podStartE2EDuration="1.670737345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:08.66935366 +0000 UTC m=+516.390432337" watchObservedRunningTime="2026-11-01 20:07:08.670737345 +0000 UTC m=+516.391816021"
Nov 01 20:07:09.242949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:09.242903    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:11.869578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.869532    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:07:11.874106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.874081    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.878623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.878599    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e3c4a7-openshift-default-dockercfg-5q4zn\""
Nov 01 20:07:11.888970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.888944    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.967864    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.967944    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.968098    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.968188    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.969629    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.969808    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.969849    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.969895    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl68g\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:11.980814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:11.969962    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071269    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071314    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071374    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071402    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071497    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071536    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071575    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl68g\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071612    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.071941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071916    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.072027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.071959    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.072027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.072014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.072138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.072113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.072425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.072402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.073584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.073562    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.073934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.073916    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.079433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.079377    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.079611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.079593    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl68g\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\") pod \"e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") " pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.192934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.192843    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:12.475375 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:12.475344    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6aafe53_3043_4cec_a3b4_8b1dfc5b5a1b.slice/crio-3e2ef6044704c035a3e3269c0860a8946c765c0854ea1ed9edfd9b838e01fa11 WatchSource:0}: Error finding container 3e2ef6044704c035a3e3269c0860a8946c765c0854ea1ed9edfd9b838e01fa11: Status 404 returned error can't find the container with id 3e2ef6044704c035a3e3269c0860a8946c765c0854ea1ed9edfd9b838e01fa11
Nov 01 20:07:12.475623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.475592    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:07:12.477742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.477713    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:12.477825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.477813    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:12.477868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.477850    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:12.686520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.686481    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerStarted","Data":"feae474dd2e5f66d690605410807ae6f4587cd72931aa13abaf1d4be70ab1060"}
Nov 01 20:07:12.686697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.686549    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerStarted","Data":"3e2ef6044704c035a3e3269c0860a8946c765c0854ea1ed9edfd9b838e01fa11"}
Nov 01 20:07:12.709018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.708959    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podStartSLOduration=1.708943235 podStartE2EDuration="1.708943235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:12.708029324 +0000 UTC m=+520.429108001" watchObservedRunningTime="2026-11-01 20:07:12.708943235 +0000 UTC m=+520.430021911"
Nov 01 20:07:12.812240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.812194    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:12.824602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:12.824578    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:13.193076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:13.192993    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:13.198497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:13.198475    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:13.692640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:13.692607    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:13.693585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:13.693565    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:13.704432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:13.704408    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:07:18.242307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:18.242270    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:18.254943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:18.254917    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:18.737100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:18.737071    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:07:28.602807 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:07:28.602754    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:07:30.576256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.576220    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:07:30.583745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.583714    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.591233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.591200    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4a127b-openshift-default-dockercfg-hmj9h\""
Nov 01 20:07:30.591387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.591371    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707269    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707359    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gwk\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707394    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707546    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707585    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707644    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.710875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.707706    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.808875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.808843    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gwk\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.808875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.808878    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.808901    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809027    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809074    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809187    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809222    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809256    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809277    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809439    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809553    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809617    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.809942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.809919    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.811712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.811686    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.811996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.811979    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.817019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.816999    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.817105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.817035    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gwk\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\") pod \"e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") " pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:30.912219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:30.912142    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:31.228528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.228501    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:07:31.229379 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:31.229352    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15527bc2_40d5_4af4_9eb0_332391978340.slice/crio-5d9eef236a42b926fd108869f1591e779e75800f3e02e57f972ab2b6096b07cd WatchSource:0}: Error finding container 5d9eef236a42b926fd108869f1591e779e75800f3e02e57f972ab2b6096b07cd: Status 404 returned error can't find the container with id 5d9eef236a42b926fd108869f1591e779e75800f3e02e57f972ab2b6096b07cd
Nov 01 20:07:31.232241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.232207    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:31.232340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.232271    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:31.232340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.232303    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:31.803304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.803268    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerStarted","Data":"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"}
Nov 01 20:07:31.803752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.803329    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerStarted","Data":"5d9eef236a42b926fd108869f1591e779e75800f3e02e57f972ab2b6096b07cd"}
Nov 01 20:07:31.827379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.827331    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podStartSLOduration=1.82731746 podStartE2EDuration="1.82731746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:31.824198611 +0000 UTC m=+539.545277301" watchObservedRunningTime="2026-11-01 20:07:31.82731746 +0000 UTC m=+539.548396183"
Nov 01 20:07:31.913241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:31.913204    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:32.918109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:32.918066    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:33.815176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:33.815138    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:33.816015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:33.815995    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:38.852624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.852591    5041 generic.go:356] "Generic (PLEG): container finished" podID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerID="feae474dd2e5f66d690605410807ae6f4587cd72931aa13abaf1d4be70ab1060" exitCode=0
Nov 01 20:07:38.853086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.852671    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerDied","Data":"feae474dd2e5f66d690605410807ae6f4587cd72931aa13abaf1d4be70ab1060"}
Nov 01 20:07:38.853132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.853116    5041 scope.go:122] "RemoveContainer" containerID="feae474dd2e5f66d690605410807ae6f4587cd72931aa13abaf1d4be70ab1060"
Nov 01 20:07:38.854475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.854444    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:38.854554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.854518    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:38.854593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:38.854551    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:39.876993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:39.876955    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerStarted","Data":"5ecbee747b49e4f30e2af5d47f7c75a81bc0e8f10198cb49eafc5a216c3a263c"}
Nov 01 20:07:40.193875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.193691    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:40.204453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.204419    5041 patch_prober.go:29] interesting pod/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 20:07:40.204708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.204674    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 20:07:40.874880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.874843    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:07:40.913413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.913381    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:07:40.913798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.913497    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:40.917904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.917866    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\""
Nov 01 20:07:40.918572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.918541    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"]
Nov 01 20:07:40.923206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.922193    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:40.927745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.927719    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"]
Nov 01 20:07:40.934588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.934563    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:07:40.935177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:40.934812    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:07:41.017470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.017435    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:07:41.017470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.017459    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:07:41.078802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.077872    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:07:41.082268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.082240    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.093759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.093730    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:07:41.108798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.108751    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.108948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.108864    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jft8n\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.108959    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.108991    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109150    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109179    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkzv\" (UniqueName: \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.109337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109268    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109294    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109317    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109357    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109386    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109444    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.109499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109541    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.109615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109579    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109605    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109629    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.109685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109662    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109695    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrdb\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.109812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.109758    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.117507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.117471    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e27ae7-openshift-default-dockercfg-855jv\""
Nov 01 20:07:41.198864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.196914    5041 patch_prober.go:29] interesting pod/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 20:07:41.198864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.196977    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.211847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jft8n\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.211921    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.211940    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.211965    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.211982    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212003    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkzv\" (UniqueName: \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212061    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212086    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212115    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212151    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212180    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212225    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212311    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212347    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212375    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212398    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212435    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212444    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212467    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrdb\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212562    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212692    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.212803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212698    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.212882    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.213316    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.213335    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.213567    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.213848    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.214026    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.214211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.214044    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.214570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.214283    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.216355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.216306    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.216483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.216446    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.217144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.217097    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.217555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.217517    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.219317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.219187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.221687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.221353    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jft8n\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.223699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.223674    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrdb\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.224663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.224445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.224845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.224814    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.225433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.225409    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkzv\" (UniqueName: \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\") pod \"maas-api-revoke-sub-e6a66eec-ghdtq\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.235940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.235918    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:41.281793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.281750    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:41.400413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.400384    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:41.574395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.574367    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:07:41.578546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.578470    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"]
Nov 01 20:07:41.579931 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:41.579903    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94f02de_c02d_419b_a4e4_422a55ad2bba.slice/crio-b23ea4c0554691afdf08836def8cd8fac4606ae17179e01e957704e2c269f59a WatchSource:0}: Error finding container b23ea4c0554691afdf08836def8cd8fac4606ae17179e01e957704e2c269f59a: Status 404 returned error can't find the container with id b23ea4c0554691afdf08836def8cd8fac4606ae17179e01e957704e2c269f59a
Nov 01 20:07:41.584451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.584412    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.584558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.584507    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.584617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.584552    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.711646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.711621    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:07:41.772810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.772704    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.772966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.772887    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.772966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.772931    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:41.928339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.928302    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq" event={"ID":"e09278f6-f095-4a02-a3ab-08f3f4a21bcf","Type":"ContainerStarted","Data":"5f187e0e8551a00d6497285a79c519bf9d618117bf4497a24eb12a459827ba20"}
Nov 01 20:07:41.930086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.930050    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" event={"ID":"0867667c-ce38-4041-8ee5-a935a60d9401","Type":"ContainerStarted","Data":"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"}
Nov 01 20:07:41.930232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.930105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" event={"ID":"0867667c-ce38-4041-8ee5-a935a60d9401","Type":"ContainerStarted","Data":"742657103c432d1f7ac32c4dbe7492bb05b9d65fb3a34ea006cecaf533181f04"}
Nov 01 20:07:41.931904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.931880    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" event={"ID":"b94f02de-c02d-419b-a4e4-422a55ad2bba","Type":"ContainerStarted","Data":"93f60a129f803eefdeb24d642577a65b15147f3c79603294da991e4ad83c50bc"}
Nov 01 20:07:41.932016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.931917    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" event={"ID":"b94f02de-c02d-419b-a4e4-422a55ad2bba","Type":"ContainerStarted","Data":"b23ea4c0554691afdf08836def8cd8fac4606ae17179e01e957704e2c269f59a"}
Nov 01 20:07:41.985964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.985903    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podStartSLOduration=0.985883493 podStartE2EDuration="985.883493ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:41.959091507 +0000 UTC m=+549.680170184" watchObservedRunningTime="2026-11-01 20:07:41.985883493 +0000 UTC m=+549.706962171"
Nov 01 20:07:41.987211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:41.987168    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podStartSLOduration=1.987158287 podStartE2EDuration="1.987158287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:41.983182209 +0000 UTC m=+549.704260886" watchObservedRunningTime="2026-11-01 20:07:41.987158287 +0000 UTC m=+549.708236962"
Nov 01 20:07:42.198806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.198180    5041 patch_prober.go:29] interesting pod/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 20:07:42.198806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.198244    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 20:07:42.236552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.236396    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:42.238114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.238092    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:42.238232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.238145    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:42.403731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.403108    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:42.405414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.405355    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:42.405565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.405433    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:42.990860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.989803    5041 generic.go:356] "Generic (PLEG): container finished" podID="e09278f6-f095-4a02-a3ab-08f3f4a21bcf" containerID="51b66fed496f0b5bca8566df493e36e99ef91676e5e43a59caa3fffccf14af2d" exitCode=0
Nov 01 20:07:42.991336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:42.991063    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq" event={"ID":"e09278f6-f095-4a02-a3ab-08f3f4a21bcf","Type":"ContainerDied","Data":"51b66fed496f0b5bca8566df493e36e99ef91676e5e43a59caa3fffccf14af2d"}
Nov 01 20:07:43.193802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.193516    5041 patch_prober.go:29] interesting pod/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 20:07:43.193802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.193581    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 20:07:43.258871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.250876    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:43.258871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.250938    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:43.407812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.404249    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:43.407812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.404315    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:43.693950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:43.693908    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:44.227804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.217119    5041 patch_prober.go:29] interesting pod/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 01 20:07:44.227804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.217189    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 01 20:07:44.240803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.238571    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:44.240803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.238642    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:44.404195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.404147    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:44.404437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.404408    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:44.793829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:44.792968    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:07:45.099797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.097395    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq" event={"ID":"e09278f6-f095-4a02-a3ab-08f3f4a21bcf","Type":"ContainerDied","Data":"5f187e0e8551a00d6497285a79c519bf9d618117bf4497a24eb12a459827ba20"}
Nov 01 20:07:45.099797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.097464    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f187e0e8551a00d6497285a79c519bf9d618117bf4497a24eb12a459827ba20"
Nov 01 20:07:45.106977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.106944    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:45.173805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.157501    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\") pod \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") "
Nov 01 20:07:45.191398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.181672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca" pod "e09278f6-f095-4a02-a3ab-08f3f4a21bcf" (UID: "e09278f6-f095-4a02-a3ab-08f3f4a21bcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:07:45.191749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.191726    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\" (UniqueName: \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\") pod \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") "
Nov 01 20:07:45.200235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.200208    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\") pod \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\" (UID: \"e09278f6-f095-4a02-a3ab-08f3f4a21bcf\") "
Nov 01 20:07:45.237486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.220893    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca" pod "e09278f6-f095-4a02-a3ab-08f3f4a21bcf" (UID: "e09278f6-f095-4a02-a3ab-08f3f4a21bcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:07:45.237946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.237922    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:45.238012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.237976    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:45.238393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.238369    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:45.239026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.238996    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:45.280504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.280460    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv" pod "e09278f6-f095-4a02-a3ab-08f3f4a21bcf" (UID: "e09278f6-f095-4a02-a3ab-08f3f4a21bcf"). InnerVolumeSpecName "kube-api-access-6vkzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:07:45.372866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.366880    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkzv\" (UniqueName: \"kubernetes.io/projected/e09278f6-f095-4a02-a3ab-08f3f4a21bcf-kube-api-access-6vkzv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:45.406946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.406893    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:45.407138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.406960    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:45.564375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.564339    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:45.636860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:45.636738    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:07:46.124804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:46.121632    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"
Nov 01 20:07:46.237319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:46.237236    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:46.237457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:46.237307    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:46.406226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:46.406187    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:46.406850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:46.406815    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:47.236510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:47.236479    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:47.236747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:47.236725    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:47.405804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:47.405347    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:47.405804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:47.405536    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:48.238813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:48.237186    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:48.238813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:48.237268    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:48.417799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:48.412298    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:48.417799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:48.412361    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:49.237255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.237218    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Nov 01 20:07:49.237424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.237285    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Nov 01 20:07:49.406239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.406197    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Nov 01 20:07:49.406879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.406269    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Nov 01 20:07:49.859797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.857934    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:07:49.859797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:49.858011    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:07:50.241339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.241258    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:50.241666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.241642    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:50.242336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.242314    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:07:50.347920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.347887    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"]
Nov 01 20:07:50.348474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.348454    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e09278f6-f095-4a02-a3ab-08f3f4a21bcf" containerName="revoke-keys"
Nov 01 20:07:50.348474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.348472    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09278f6-f095-4a02-a3ab-08f3f4a21bcf" containerName="revoke-keys"
Nov 01 20:07:50.348642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.348594    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09278f6-f095-4a02-a3ab-08f3f4a21bcf" containerName="revoke-keys"
Nov 01 20:07:50.353178 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.353156    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.358947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.358923    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:07:50.359112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.358933    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:07:50.359216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.359012    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:07:50.359579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.358923    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:07:50.360276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.360253    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"]
Nov 01 20:07:50.405910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.405884    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:50.406326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.406308    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:50.407062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.407043    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:07:50.447879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.447843    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.448644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.448617    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.449021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.448999    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjctp\" (UniqueName: \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.550041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.550006    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.550235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.550066    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.550235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.550135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjctp\" (UniqueName: \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.550822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.550760    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.552440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.552416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.558892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.558868    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjctp\" (UniqueName: \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\") pod \"maas-api-revoke-sub-499b54cc-rm4z8\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.671228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.671190    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:50.951311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:50.951235    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"]
Nov 01 20:07:50.955544 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:07:50.955508    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd72ffa_1ecb_446e_9c62_17f3158ee0a5.slice/crio-200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb WatchSource:0}: Error finding container 200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb: Status 404 returned error can't find the container with id 200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb
Nov 01 20:07:51.188574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:51.188537    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8" event={"ID":"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5","Type":"ContainerStarted","Data":"200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb"}
Nov 01 20:07:52.197564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.197462    5041 generic.go:356] "Generic (PLEG): container finished" podID="fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" containerID="3ce99d9c5a5b7bbb20236ea6162fde975350586d71e02ff153599d9546a721b0" exitCode=0
Nov 01 20:07:52.197564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.197548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8" event={"ID":"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5","Type":"ContainerDied","Data":"3ce99d9c5a5b7bbb20236ea6162fde975350586d71e02ff153599d9546a721b0"}
Nov 01 20:07:52.199511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.199485    5041 generic.go:356] "Generic (PLEG): container finished" podID="15527bc2-40d5-4af4-9eb0-332391978340" containerID="d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d" exitCode=0
Nov 01 20:07:52.199642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.199570    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerDied","Data":"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"}
Nov 01 20:07:52.200296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.200272    5041 scope.go:122] "RemoveContainer" containerID="d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"
Nov 01 20:07:52.202390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.202323    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:52.202568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.202546    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:52.202669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:52.202651    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:07:53.222347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.222304    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerStarted","Data":"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"}
Nov 01 20:07:53.626533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.626500    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:53.812594 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.812563    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\") pod \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") "
Nov 01 20:07:53.812770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.812679    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\") pod \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") "
Nov 01 20:07:53.812770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.812732    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\" (UniqueName: \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\") pod \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\" (UID: \"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5\") "
Nov 01 20:07:53.813091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.813065    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca" pod "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" (UID: "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:07:53.813276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.813244    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:53.814600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.814576    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca" pod "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" (UID: "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:07:53.814886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.814863    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp" pod "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" (UID: "fbd72ffa-1ecb-446e-9c62-17f3158ee0a5"). InnerVolumeSpecName "kube-api-access-xjctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:07:53.912731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.912648    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:53.912932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.912841    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:07:53.912932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.912909    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:07:53.914473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.914452    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjctp\" (UniqueName: \"kubernetes.io/projected/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-kube-api-access-xjctp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:53.914473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:53.914473    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:54.229711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.229627    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"
Nov 01 20:07:54.229711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.229664    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8" event={"ID":"fbd72ffa-1ecb-446e-9c62-17f3158ee0a5","Type":"ContainerDied","Data":"200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb"}
Nov 01 20:07:54.230264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.229724    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200f158971c977979dde4e9afcc0a571959c69c18e73c3bada3e2e4cab254edb"
Nov 01 20:07:54.622694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.622661    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"]
Nov 01 20:07:54.623329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.623306    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" containerName="revoke-keys"
Nov 01 20:07:54.623329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.623329    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" containerName="revoke-keys"
Nov 01 20:07:54.623497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.623471    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" containerName="revoke-keys"
Nov 01 20:07:54.625804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.625769    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.635939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.635914    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"]
Nov 01 20:07:54.710827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.710795    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:07:54.711062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.711043    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:07:54.725516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.725493    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:07:54.725516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.725503    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:07:54.726018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.725995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2kz\" (UniqueName: \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.726108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.726059    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.726194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.726176    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.827513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.827477    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2kz\" (UniqueName: \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.827719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.827535    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.827719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.827596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.828320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.828299    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.830579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.830548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.835390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.835360    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2kz\" (UniqueName: \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\") pod \"maas-api-revoke-sub-eab37f75-lknbk\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:54.913209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.913128    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:07:54.913348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.913207    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:07:54.939437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:54.939409    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:55.276395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:55.276366    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"]
Nov 01 20:07:55.913341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:55.913305    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:07:55.913519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:55.913358    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:07:56.271756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:56.271716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk" event={"ID":"59401097-6dc5-4ef1-9b01-f29a3dc81f9f","Type":"ContainerStarted","Data":"d6b463c3afc27ab83502299a0ed1283b93ed9e0fc87de705e91c5ed729400cae"}
Nov 01 20:07:56.271986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:56.271802    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk" event={"ID":"59401097-6dc5-4ef1-9b01-f29a3dc81f9f","Type":"ContainerStarted","Data":"d70b8d9da4434cba5193dffbb4ccc83c7e01f9a8aaf2a82180085356a51f47d9"}
Nov 01 20:07:56.289068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:56.289017    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk" podStartSLOduration=2.289001827 podStartE2EDuration="2.289001827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:56.287077039 +0000 UTC m=+564.008155715" watchObservedRunningTime="2026-11-01 20:07:56.289001827 +0000 UTC m=+564.010080502"
Nov 01 20:07:56.913307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:56.913268    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:07:56.913518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:56.913345    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:07:57.280264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:57.280180    5041 generic.go:356] "Generic (PLEG): container finished" podID="59401097-6dc5-4ef1-9b01-f29a3dc81f9f" containerID="d6b463c3afc27ab83502299a0ed1283b93ed9e0fc87de705e91c5ed729400cae" exitCode=0
Nov 01 20:07:57.280419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:57.280271    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk" event={"ID":"59401097-6dc5-4ef1-9b01-f29a3dc81f9f","Type":"ContainerDied","Data":"d6b463c3afc27ab83502299a0ed1283b93ed9e0fc87de705e91c5ed729400cae"}
Nov 01 20:07:57.913473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:57.913440    5041 patch_prober.go:29] interesting pod/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 01 20:07:57.913891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:57.913498    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 01 20:07:58.521084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.520740    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:58.572880 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:07:58.569229    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:07:58.616331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.616292    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\") pod \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") "
Nov 01 20:07:58.616514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.616351    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\") pod \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") "
Nov 01 20:07:58.616514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.616391    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\" (UniqueName: \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\") pod \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\" (UID: \"59401097-6dc5-4ef1-9b01-f29a3dc81f9f\") "
Nov 01 20:07:58.617086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.617040    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca" pod "59401097-6dc5-4ef1-9b01-f29a3dc81f9f" (UID: "59401097-6dc5-4ef1-9b01-f29a3dc81f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:07:58.619349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.619323    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz" pod "59401097-6dc5-4ef1-9b01-f29a3dc81f9f" (UID: "59401097-6dc5-4ef1-9b01-f29a3dc81f9f"). InnerVolumeSpecName "kube-api-access-ms2kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:07:58.662935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.662873    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca" pod "59401097-6dc5-4ef1-9b01-f29a3dc81f9f" (UID: "59401097-6dc5-4ef1-9b01-f29a3dc81f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:07:58.719150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.719114    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:58.719150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.719145    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:58.719150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.719156    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2kz\" (UniqueName: \"kubernetes.io/projected/59401097-6dc5-4ef1-9b01-f29a3dc81f9f-kube-api-access-ms2kz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:07:58.951231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.951119    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:58.953456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.953435    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:58.960207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:58.960182    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:07:59.318804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:59.316736    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"
Nov 01 20:07:59.327533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:59.326058    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk" event={"ID":"59401097-6dc5-4ef1-9b01-f29a3dc81f9f","Type":"ContainerDied","Data":"d70b8d9da4434cba5193dffbb4ccc83c7e01f9a8aaf2a82180085356a51f47d9"}
Nov 01 20:07:59.327533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:07:59.326112    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70b8d9da4434cba5193dffbb4ccc83c7e01f9a8aaf2a82180085356a51f47d9"
Nov 01 20:08:04.893110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.893078    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"]
Nov 01 20:08:04.893852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.893832    5041 cpu_manager.go:414] "containerMap: removing container" podUID="59401097-6dc5-4ef1-9b01-f29a3dc81f9f" containerName="revoke-keys"
Nov 01 20:08:04.893963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.893854    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59401097-6dc5-4ef1-9b01-f29a3dc81f9f" containerName="revoke-keys"
Nov 01 20:08:04.894023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.894001    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="59401097-6dc5-4ef1-9b01-f29a3dc81f9f" containerName="revoke-keys"
Nov 01 20:08:04.896704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.896683    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:04.900419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.900384    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:04.900522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.900425    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:04.901828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.901808    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:04.901942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.901830    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:04.906097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:04.905712    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"]
Nov 01 20:08:05.001107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.001073    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv478\" (UniqueName: \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.001281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.001140    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.001281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.001203    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.102328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.102295    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv478\" (UniqueName: \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.102503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.102354    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.102503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.102378    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.103121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.103095    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.104846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.104828    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.109706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.109684    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv478\" (UniqueName: \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\") pod \"maas-api-revoke-sub-7fa76a8d-rjvnh\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.213810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.213456    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:05.498819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:05.494430    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"]
Nov 01 20:08:05.504644 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:05.504615    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac1d6c0_ccd7_4191_abc4_bd753c46e9bf.slice/crio-c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086 WatchSource:0}: Error finding container c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086: Status 404 returned error can't find the container with id c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086
Nov 01 20:08:06.383233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:06.383191    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh" event={"ID":"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf","Type":"ContainerStarted","Data":"e835c1d67208b9d9b4d2c5f65d36bfe3dfd980bcf87fb538a4d37bafd04e95b7"}
Nov 01 20:08:06.383870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:06.383262    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh" event={"ID":"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf","Type":"ContainerStarted","Data":"c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086"}
Nov 01 20:08:06.402074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:06.402022    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh" podStartSLOduration=2.402008511 podStartE2EDuration="2.402008511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:06.399728855 +0000 UTC m=+574.120807532" watchObservedRunningTime="2026-11-01 20:08:06.402008511 +0000 UTC m=+574.123087186"
Nov 01 20:08:07.390993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.390960    5041 generic.go:356] "Generic (PLEG): container finished" podID="3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" containerID="e835c1d67208b9d9b4d2c5f65d36bfe3dfd980bcf87fb538a4d37bafd04e95b7" exitCode=0
Nov 01 20:08:07.391398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.391046    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh" event={"ID":"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf","Type":"ContainerDied","Data":"e835c1d67208b9d9b4d2c5f65d36bfe3dfd980bcf87fb538a4d37bafd04e95b7"}
Nov 01 20:08:07.471729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.471693    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"]
Nov 01 20:08:07.474947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.474927    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.482713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.482686    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"]
Nov 01 20:08:07.535888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.535852    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.536057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.535906    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.536097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.536073    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghr2\" (UniqueName: \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.637483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.637445    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.637667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.637502    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.637667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.637597    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghr2\" (UniqueName: \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.638241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.638217    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.640033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.640011    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.645608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.645545    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghr2\" (UniqueName: \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\") pod \"maas-api-revoke-sub-95b2faa7-9qf64\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:07.794788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:07.794749    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:08.096051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.096027    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"]
Nov 01 20:08:08.096813 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:08.096762    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7ff6c6_93c3_4b0a_8cbb_9cfb252e6371.slice/crio-ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949 WatchSource:0}: Error finding container ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949: Status 404 returned error can't find the container with id ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949
Nov 01 20:08:08.398984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.398950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64" event={"ID":"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371","Type":"ContainerStarted","Data":"4b7a147060d9743e4b7a25b2d6748550e4cdf4384c69f97a8930da286f52b4f5"}
Nov 01 20:08:08.399458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.399014    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64" event={"ID":"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371","Type":"ContainerStarted","Data":"ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949"}
Nov 01 20:08:08.418230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.418181    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64" podStartSLOduration=1.418167746 podStartE2EDuration="1.418167746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:08.415607499 +0000 UTC m=+576.136686176" watchObservedRunningTime="2026-11-01 20:08:08.418167746 +0000 UTC m=+576.139246425"
Nov 01 20:08:08.544385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.544356    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:08.648511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.648427    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\" (UniqueName: \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\") pod \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") "
Nov 01 20:08:08.648511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.648479    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\") pod \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") "
Nov 01 20:08:08.648745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.648536    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\") pod \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\" (UID: \"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf\") "
Nov 01 20:08:08.649005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.648972    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca" pod "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" (UID: "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:08.650670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.650646    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478" pod "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" (UID: "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf"). InnerVolumeSpecName "kube-api-access-cv478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:08.650896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.650865    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca" pod "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" (UID: "3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:08.750488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.750456    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:08.750488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.750484    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv478\" (UniqueName: \"kubernetes.io/projected/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-kube-api-access-cv478\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:08.750488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:08.750495    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:09.407725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:09.407688    5041 generic.go:356] "Generic (PLEG): container finished" podID="7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" containerID="4b7a147060d9743e4b7a25b2d6748550e4cdf4384c69f97a8930da286f52b4f5" exitCode=0
Nov 01 20:08:09.408199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:09.407771    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64" event={"ID":"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371","Type":"ContainerDied","Data":"4b7a147060d9743e4b7a25b2d6748550e4cdf4384c69f97a8930da286f52b4f5"}
Nov 01 20:08:09.409265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:09.409247    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"
Nov 01 20:08:09.409375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:09.409276    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh" event={"ID":"3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf","Type":"ContainerDied","Data":"c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086"}
Nov 01 20:08:09.409375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:09.409318    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d5b8442f221ea0bd33c257d2c2056ec97d4d50725aaa4d9811fac6518d6086"
Nov 01 20:08:10.610304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.610277    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:10.676361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.676277    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\") pod \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") "
Nov 01 20:08:10.676552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.676529    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\") pod \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") "
Nov 01 20:08:10.676604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.676587    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\" (UniqueName: \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\") pod \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\" (UID: \"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371\") "
Nov 01 20:08:10.676666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.676648    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca" pod "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" (UID: "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:10.677105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.677081    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:10.678637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.678609    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca" pod "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" (UID: "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:10.678748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.678672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2" pod "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" (UID: "7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371"). InnerVolumeSpecName "kube-api-access-jghr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:10.778243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.778193    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:10.778243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:10.778230    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghr2\" (UniqueName: \"kubernetes.io/projected/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371-kube-api-access-jghr2\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:11.424374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:11.424341    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"
Nov 01 20:08:11.424374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:11.424358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64" event={"ID":"7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371","Type":"ContainerDied","Data":"ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949"}
Nov 01 20:08:11.424639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:11.424420    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7efe64cc433f642176383d987f7165ad78297df7062e138fb87905b2ca7949"
Nov 01 20:08:17.521074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.520987    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"]
Nov 01 20:08:17.521625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521608    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" containerName="revoke-keys"
Nov 01 20:08:17.521678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521625    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" containerName="revoke-keys"
Nov 01 20:08:17.521678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521653    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" containerName="revoke-keys"
Nov 01 20:08:17.521678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521659    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" containerName="revoke-keys"
Nov 01 20:08:17.521770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521748    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" containerName="revoke-keys"
Nov 01 20:08:17.521770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.521761    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" containerName="revoke-keys"
Nov 01 20:08:17.524865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.524850    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.529574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.529456    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:17.529574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.529523    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:17.529821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.529609    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:17.530006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.529981    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:17.532741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.532715    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"]
Nov 01 20:08:17.577420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.577390    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.577620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.577441    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt9kg\" (UniqueName: \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.577620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.577563    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.678736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.678698    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.678943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.678813    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt9kg\" (UniqueName: \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.678943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.678871    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.679573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.679547    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.681124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.681102    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.687183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.687161    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt9kg\" (UniqueName: \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\") pod \"maas-api-revoke-sub-52ed9af5-s5dks\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:17.841311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:17.841273    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:18.123677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:18.123645    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"]
Nov 01 20:08:18.124421 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:18.124395    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb7410e_b652_4eb6_a590_a466be89d726.slice/crio-d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3 WatchSource:0}: Error finding container d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3: Status 404 returned error can't find the container with id d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3
Nov 01 20:08:18.584916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:18.584886    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks" event={"ID":"ccb7410e-b652-4eb6-a590-a466be89d726","Type":"ContainerStarted","Data":"1c1e73c6227dedc214a55fa457784f62e2978214ee664d8795ae461c4e7777cc"}
Nov 01 20:08:18.585257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:18.584945    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks" event={"ID":"ccb7410e-b652-4eb6-a590-a466be89d726","Type":"ContainerStarted","Data":"d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3"}
Nov 01 20:08:18.600975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:18.600928    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks" podStartSLOduration=1.600914204 podStartE2EDuration="1.600914204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:18.59901342 +0000 UTC m=+586.320092096" watchObservedRunningTime="2026-11-01 20:08:18.600914204 +0000 UTC m=+586.321992879"
Nov 01 20:08:19.593852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:19.593818    5041 generic.go:356] "Generic (PLEG): container finished" podID="ccb7410e-b652-4eb6-a590-a466be89d726" containerID="1c1e73c6227dedc214a55fa457784f62e2978214ee664d8795ae461c4e7777cc" exitCode=0
Nov 01 20:08:19.594297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:19.593892    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks" event={"ID":"ccb7410e-b652-4eb6-a590-a466be89d726","Type":"ContainerDied","Data":"1c1e73c6227dedc214a55fa457784f62e2978214ee664d8795ae461c4e7777cc"}
Nov 01 20:08:20.765114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.765085    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:20.818978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.818942    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\") pod \"ccb7410e-b652-4eb6-a590-a466be89d726\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") "
Nov 01 20:08:20.819161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.818999    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\") pod \"ccb7410e-b652-4eb6-a590-a466be89d726\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") "
Nov 01 20:08:20.819237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.819166    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\" (UniqueName: \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\") pod \"ccb7410e-b652-4eb6-a590-a466be89d726\" (UID: \"ccb7410e-b652-4eb6-a590-a466be89d726\") "
Nov 01 20:08:20.819340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.819315    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca" pod "ccb7410e-b652-4eb6-a590-a466be89d726" (UID: "ccb7410e-b652-4eb6-a590-a466be89d726"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:20.819671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.819648    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:20.821653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.821632    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca" pod "ccb7410e-b652-4eb6-a590-a466be89d726" (UID: "ccb7410e-b652-4eb6-a590-a466be89d726"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:20.821743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.821652    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg" pod "ccb7410e-b652-4eb6-a590-a466be89d726" (UID: "ccb7410e-b652-4eb6-a590-a466be89d726"). InnerVolumeSpecName "kube-api-access-qt9kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:20.920570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.920496    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb7410e-b652-4eb6-a590-a466be89d726-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:20.920570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:20.920525    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt9kg\" (UniqueName: \"kubernetes.io/projected/ccb7410e-b652-4eb6-a590-a466be89d726-kube-api-access-qt9kg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:21.608519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:21.608486    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"
Nov 01 20:08:21.608679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:21.608518    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks" event={"ID":"ccb7410e-b652-4eb6-a590-a466be89d726","Type":"ContainerDied","Data":"d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3"}
Nov 01 20:08:21.608679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:21.608568    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58252bdbfac6e2fdd57b7cd28e2b94f6af49408ed9e7b92e6e8114d0fc6d0f3"
Nov 01 20:08:22.201994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.201959    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"]
Nov 01 20:08:22.202892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.202871    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ccb7410e-b652-4eb6-a590-a466be89d726" containerName="revoke-keys"
Nov 01 20:08:22.202945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.202896    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb7410e-b652-4eb6-a590-a466be89d726" containerName="revoke-keys"
Nov 01 20:08:22.203109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.203095    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb7410e-b652-4eb6-a590-a466be89d726" containerName="revoke-keys"
Nov 01 20:08:22.209487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.209462    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.212060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.212033    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"]
Nov 01 20:08:22.214235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.214168    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:22.214235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.214207    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:22.214235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.214237    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:22.214578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.214422    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:22.340755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.340721    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.340941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.340766    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5b6\" (UniqueName: \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.340993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.340945    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.446232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.446196    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.446416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.446258    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5b6\" (UniqueName: \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.446416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.446315    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.447029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.447003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.448585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.448560    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.454135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.454081    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5b6\" (UniqueName: \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\") pod \"maas-api-revoke-sub-ce4ecaa5-mkq68\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.525227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.525189    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:22.813040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:22.813007    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"]
Nov 01 20:08:22.814488 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:22.814457    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6fb124_d3be_4b60_89a6_e3134f94ad59.slice/crio-b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd WatchSource:0}: Error finding container b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd: Status 404 returned error can't find the container with id b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd
Nov 01 20:08:23.640654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:23.640596    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68" event={"ID":"6b6fb124-d3be-4b60-89a6-e3134f94ad59","Type":"ContainerStarted","Data":"b422390e6059ab49eade7f9e861645dbac8d7841c5feaea86d4369c2fce903b0"}
Nov 01 20:08:23.641202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:23.640683    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68" event={"ID":"6b6fb124-d3be-4b60-89a6-e3134f94ad59","Type":"ContainerStarted","Data":"b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd"}
Nov 01 20:08:23.656662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:23.656612    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68" podStartSLOduration=1.656597882 podStartE2EDuration="1.656597882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:23.655280933 +0000 UTC m=+591.376359610" watchObservedRunningTime="2026-11-01 20:08:23.656597882 +0000 UTC m=+591.377676659"
Nov 01 20:08:24.648136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:24.648100    5041 generic.go:356] "Generic (PLEG): container finished" podID="6b6fb124-d3be-4b60-89a6-e3134f94ad59" containerID="b422390e6059ab49eade7f9e861645dbac8d7841c5feaea86d4369c2fce903b0" exitCode=0
Nov 01 20:08:24.648577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:24.648186    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68" event={"ID":"6b6fb124-d3be-4b60-89a6-e3134f94ad59","Type":"ContainerDied","Data":"b422390e6059ab49eade7f9e861645dbac8d7841c5feaea86d4369c2fce903b0"}
Nov 01 20:08:25.805289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.805263    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:25.885726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.885688    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\") pod \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") "
Nov 01 20:08:25.885938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.885869    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\" (UniqueName: \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\") pod \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") "
Nov 01 20:08:25.886030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.885990    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\") pod \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\" (UID: \"6b6fb124-d3be-4b60-89a6-e3134f94ad59\") "
Nov 01 20:08:25.886424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.886395    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca" pod "6b6fb124-d3be-4b60-89a6-e3134f94ad59" (UID: "6b6fb124-d3be-4b60-89a6-e3134f94ad59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:25.891816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.888186    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6" pod "6b6fb124-d3be-4b60-89a6-e3134f94ad59" (UID: "6b6fb124-d3be-4b60-89a6-e3134f94ad59"). InnerVolumeSpecName "kube-api-access-4t5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:25.891816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.888497    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca" pod "6b6fb124-d3be-4b60-89a6-e3134f94ad59" (UID: "6b6fb124-d3be-4b60-89a6-e3134f94ad59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:25.988212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.988115    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:25.988212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.988159    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5b6\" (UniqueName: \"kubernetes.io/projected/6b6fb124-d3be-4b60-89a6-e3134f94ad59-kube-api-access-4t5b6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:25.988212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:25.988171    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fb124-d3be-4b60-89a6-e3134f94ad59-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:26.050678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.050643    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:08:26.051557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.051535    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6b6fb124-d3be-4b60-89a6-e3134f94ad59" containerName="revoke-keys"
Nov 01 20:08:26.051612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.051562    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6fb124-d3be-4b60-89a6-e3134f94ad59" containerName="revoke-keys"
Nov 01 20:08:26.051731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.051718    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6fb124-d3be-4b60-89a6-e3134f94ad59" containerName="revoke-keys"
Nov 01 20:08:26.055825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.055805    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.069106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.069075    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:08:26.190084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190298    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcwd\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190357    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190393    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190450    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190519    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.190575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.190549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291619    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291670    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291727    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291755    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcwd\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291815    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.291865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291858    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.291936    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292003    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292056    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292215    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292612    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.292914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.292892    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.293411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.293386    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.294942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.294916    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.295241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.295216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.300089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.300039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.300319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.300297    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcwd\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.378494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.378463    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:26.663700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.663619    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68" event={"ID":"6b6fb124-d3be-4b60-89a6-e3134f94ad59","Type":"ContainerDied","Data":"b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd"}
Nov 01 20:08:26.663700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.663646    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"
Nov 01 20:08:26.663700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.663676    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e2fc2fad310fd9edd60c3bbf83b864af9653424ca207c82b20a1db8223b6dd"
Nov 01 20:08:26.673599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.673575    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:08:26.674281 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:26.674260    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0fa21f_1a0c_4490_809d_2640b709504d.slice/crio-da032f514ab4ab383f4b269f93486ea4de933884a8148e125d8f123649be0867 WatchSource:0}: Error finding container da032f514ab4ab383f4b269f93486ea4de933884a8148e125d8f123649be0867: Status 404 returned error can't find the container with id da032f514ab4ab383f4b269f93486ea4de933884a8148e125d8f123649be0867
Nov 01 20:08:26.676910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.676875    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:26.676990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.676951    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:26.677025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.676992    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:26.988010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.987931    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"]
Nov 01 20:08:26.993315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:26.993292    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.003879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.002353    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:27.003879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.002997    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:27.003879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.003123    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:27.003879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.003206    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:27.027469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.025909    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"]
Nov 01 20:08:27.107373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.107323    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78dd\" (UniqueName: \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.107373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.107379    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.107635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.107480    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.213526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.212670    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78dd\" (UniqueName: \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.213526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.212771    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.213526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.212862    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.221801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.217575    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.222465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.222424    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.222923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.222901    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78dd\" (UniqueName: \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\") pod \"maas-api-revoke-sub-1ee0efc2-kncss\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.332571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.332533    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:27.680223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.680121    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"]
Nov 01 20:08:27.695481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.695145    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" event={"ID":"0c0fa21f-1a0c-4490-809d-2640b709504d","Type":"ContainerStarted","Data":"c11b0459b751d25a85e7f37db496938ac8066fad37fe527384ddbddc78a2d364"}
Nov 01 20:08:27.695481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.695224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" event={"ID":"0c0fa21f-1a0c-4490-809d-2640b709504d","Type":"ContainerStarted","Data":"da032f514ab4ab383f4b269f93486ea4de933884a8148e125d8f123649be0867"}
Nov 01 20:08:27.718949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:27.718891    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podStartSLOduration=1.718872167 podStartE2EDuration="1.718872167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:27.716023769 +0000 UTC m=+595.437102445" watchObservedRunningTime="2026-11-01 20:08:27.718872167 +0000 UTC m=+595.439950844"
Nov 01 20:08:28.378725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.378643    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:28.380736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.380709    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:08:28.380866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.380756    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:08:28.574134 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:08:28.574088    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:08:28.703644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.703566    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss" event={"ID":"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77","Type":"ContainerStarted","Data":"5deca8d0c59e2dd758785842a4486987894b844449fec3b200f8964de92d475b"}
Nov 01 20:08:28.703644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.703628    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss" event={"ID":"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77","Type":"ContainerStarted","Data":"1aef343bcd8a2a34ef642aec22ae1f94ba9f980f36bedaf7aeb016249f9b290a"}
Nov 01 20:08:28.722323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:28.722273    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss" podStartSLOduration=2.7222581359999998 podStartE2EDuration="2.722258136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:28.719507924 +0000 UTC m=+596.440586599" watchObservedRunningTime="2026-11-01 20:08:28.722258136 +0000 UTC m=+596.443336826"
Nov 01 20:08:29.379726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.379687    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:08:29.380187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.379755    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:08:29.711933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.711839    5041 generic.go:356] "Generic (PLEG): container finished" podID="0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" containerID="5deca8d0c59e2dd758785842a4486987894b844449fec3b200f8964de92d475b" exitCode=0
Nov 01 20:08:29.712100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.711921    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss" event={"ID":"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77","Type":"ContainerDied","Data":"5deca8d0c59e2dd758785842a4486987894b844449fec3b200f8964de92d475b"}
Nov 01 20:08:29.874759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.874715    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"]
Nov 01 20:08:29.879282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.879256    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:29.892308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.892276    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"]
Nov 01 20:08:29.982578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.982493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:29.982734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.982676    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km28d\" (UniqueName: \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:29.982734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:29.982713    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.083484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.083454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.083644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.083581    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km28d\" (UniqueName: \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.083644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.083613    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.084442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.084412    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.085998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.085974    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.092010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.091988    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km28d\" (UniqueName: \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\") pod \"maas-api-revoke-sub-2e12204d-hfvmf\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.196207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.196169    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:30.381813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.381623    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:08:30.381813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.381691    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:08:30.504394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.504363    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"]
Nov 01 20:08:30.505085 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:30.505053    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d28eec_663d_4019_b9d2_b8ea9dec9f46.slice/crio-0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889 WatchSource:0}: Error finding container 0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889: Status 404 returned error can't find the container with id 0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889
Nov 01 20:08:30.719578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.719540    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf" event={"ID":"a6d28eec-663d-4019-b9d2-b8ea9dec9f46","Type":"ContainerStarted","Data":"0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889"}
Nov 01 20:08:30.911335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:30.911313    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:31.000912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.000878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\") pod \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") "
Nov 01 20:08:31.001097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.000967    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\" (UniqueName: \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\") pod \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") "
Nov 01 20:08:31.001183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.001168    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\") pod \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\" (UID: \"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77\") "
Nov 01 20:08:31.001552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.001523    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca" pod "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" (UID: "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:31.001673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.001639    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:31.003091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.003066    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd" pod "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" (UID: "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77"). InnerVolumeSpecName "kube-api-access-r78dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:31.003187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.003068    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca" pod "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" (UID: "0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:31.102773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.102737    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:31.102912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.102773    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78dd\" (UniqueName: \"kubernetes.io/projected/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77-kube-api-access-r78dd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:31.379030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.378945    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:08:31.379030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.379000    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:08:31.727902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.727807    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"
Nov 01 20:08:31.727902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.727821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss" event={"ID":"0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77","Type":"ContainerDied","Data":"1aef343bcd8a2a34ef642aec22ae1f94ba9f980f36bedaf7aeb016249f9b290a"}
Nov 01 20:08:31.727902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.727884    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aef343bcd8a2a34ef642aec22ae1f94ba9f980f36bedaf7aeb016249f9b290a"
Nov 01 20:08:31.729674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.729647    5041 generic.go:356] "Generic (PLEG): container finished" podID="a6d28eec-663d-4019-b9d2-b8ea9dec9f46" containerID="f5ee05f93eed72cf428ff2bd60218031efa87f2974caaf117fd275dfa61fbb53" exitCode=0
Nov 01 20:08:31.729833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:31.729730    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf" event={"ID":"a6d28eec-663d-4019-b9d2-b8ea9dec9f46","Type":"ContainerDied","Data":"f5ee05f93eed72cf428ff2bd60218031efa87f2974caaf117fd275dfa61fbb53"}
Nov 01 20:08:32.379714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:32.379677    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 01 20:08:32.379886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:32.379739    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 01 20:08:32.569681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:32.569649    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:08:32.574627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:32.574603    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:08:32.915152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:32.915127    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:33.023239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.023145    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\") pod \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") "
Nov 01 20:08:33.023373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.023258    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\" (UniqueName: \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\") pod \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") "
Nov 01 20:08:33.023430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.023395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\") pod \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\" (UID: \"a6d28eec-663d-4019-b9d2-b8ea9dec9f46\") "
Nov 01 20:08:33.023729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.023705    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca" pod "a6d28eec-663d-4019-b9d2-b8ea9dec9f46" (UID: "a6d28eec-663d-4019-b9d2-b8ea9dec9f46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:33.023931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.023914    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:33.025273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.025248    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca" pod "a6d28eec-663d-4019-b9d2-b8ea9dec9f46" (UID: "a6d28eec-663d-4019-b9d2-b8ea9dec9f46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:33.025358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.025314    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d" pod "a6d28eec-663d-4019-b9d2-b8ea9dec9f46" (UID: "a6d28eec-663d-4019-b9d2-b8ea9dec9f46"). InnerVolumeSpecName "kube-api-access-km28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:33.124865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.124830    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:33.124865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.124860    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km28d\" (UniqueName: \"kubernetes.io/projected/a6d28eec-663d-4019-b9d2-b8ea9dec9f46-kube-api-access-km28d\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:33.484335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.484221    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:33.484911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.484887    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:33.486220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.486201    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:08:33.754895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.754802    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"
Nov 01 20:08:33.754895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.754816    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf" event={"ID":"a6d28eec-663d-4019-b9d2-b8ea9dec9f46","Type":"ContainerDied","Data":"0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889"}
Nov 01 20:08:33.754895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:33.754886    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bb4b746d7d6697300d260407f53ac39da2b57b204dd99dace7c98b20558d889"
Nov 01 20:08:34.502696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.502663    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"]
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.503733    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" containerName="revoke-keys"
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.503757    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" containerName="revoke-keys"
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.503813    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a6d28eec-663d-4019-b9d2-b8ea9dec9f46" containerName="revoke-keys"
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.503821    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d28eec-663d-4019-b9d2-b8ea9dec9f46" containerName="revoke-keys"
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.503989    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d28eec-663d-4019-b9d2-b8ea9dec9f46" containerName="revoke-keys"
Nov 01 20:08:34.506237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.504012    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" containerName="revoke-keys"
Nov 01 20:08:34.508725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.508700    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.512861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.512836    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:34.514082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.514055    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"]
Nov 01 20:08:34.514224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.514209    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:34.514304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.514246    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:34.514360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.514324    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:34.645412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.645380    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.645412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.645415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsvw\" (UniqueName: \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.645600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.645453    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.746734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.746687    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.746942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.746905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.746942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.746928    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsvw\" (UniqueName: \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.747611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.747585    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.749210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.749182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.755067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.755014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsvw\" (UniqueName: \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\") pod \"maas-api-revoke-sub-fefc2af9-n6brt\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:34.827897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:34.827860    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:35.125079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:35.125049    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"]
Nov 01 20:08:35.127083 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:35.127053    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019cadb2_8af5_43b7_8854_bfcb47ef571f.slice/crio-683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32 WatchSource:0}: Error finding container 683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32: Status 404 returned error can't find the container with id 683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32
Nov 01 20:08:35.807858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:35.800329    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt" event={"ID":"019cadb2-8af5-43b7-8854-bfcb47ef571f","Type":"ContainerStarted","Data":"73283e3555ad759b4c33c8556570cc9d44112a2c77a98293c252f41ae13258ce"}
Nov 01 20:08:35.807858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:35.800404    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt" event={"ID":"019cadb2-8af5-43b7-8854-bfcb47ef571f","Type":"ContainerStarted","Data":"683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32"}
Nov 01 20:08:35.821611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:35.821558    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt" podStartSLOduration=1.821542361 podStartE2EDuration="1.821542361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:35.818832198 +0000 UTC m=+603.539910874" watchObservedRunningTime="2026-11-01 20:08:35.821542361 +0000 UTC m=+603.542621077"
Nov 01 20:08:36.809134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:36.809089    5041 generic.go:356] "Generic (PLEG): container finished" podID="019cadb2-8af5-43b7-8854-bfcb47ef571f" containerID="73283e3555ad759b4c33c8556570cc9d44112a2c77a98293c252f41ae13258ce" exitCode=0
Nov 01 20:08:36.809537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:36.809173    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt" event={"ID":"019cadb2-8af5-43b7-8854-bfcb47ef571f","Type":"ContainerDied","Data":"73283e3555ad759b4c33c8556570cc9d44112a2c77a98293c252f41ae13258ce"}
Nov 01 20:08:37.972210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:37.972183    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:38.041583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.041550    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\") pod \"019cadb2-8af5-43b7-8854-bfcb47ef571f\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") "
Nov 01 20:08:38.041740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.041622    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\" (UniqueName: \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\") pod \"019cadb2-8af5-43b7-8854-bfcb47ef571f\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") "
Nov 01 20:08:38.041740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.041645    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\") pod \"019cadb2-8af5-43b7-8854-bfcb47ef571f\" (UID: \"019cadb2-8af5-43b7-8854-bfcb47ef571f\") "
Nov 01 20:08:38.041987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.041962    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca" pod "019cadb2-8af5-43b7-8854-bfcb47ef571f" (UID: "019cadb2-8af5-43b7-8854-bfcb47ef571f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:38.042144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.042126    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:38.043847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.043820    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca" pod "019cadb2-8af5-43b7-8854-bfcb47ef571f" (UID: "019cadb2-8af5-43b7-8854-bfcb47ef571f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:38.043989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.043967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw" pod "019cadb2-8af5-43b7-8854-bfcb47ef571f" (UID: "019cadb2-8af5-43b7-8854-bfcb47ef571f"). InnerVolumeSpecName "kube-api-access-qcsvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:38.142912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.142825    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsvw\" (UniqueName: \"kubernetes.io/projected/019cadb2-8af5-43b7-8854-bfcb47ef571f-kube-api-access-qcsvw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:38.142912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.142862    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019cadb2-8af5-43b7-8854-bfcb47ef571f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:38.824630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.824597    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"
Nov 01 20:08:38.824864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.824634    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt" event={"ID":"019cadb2-8af5-43b7-8854-bfcb47ef571f","Type":"ContainerDied","Data":"683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32"}
Nov 01 20:08:38.824864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:38.824685    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="683248ad227f321e95783e69c89179f4d0768ba6936c4beaa19463894703eb32"
Nov 01 20:08:40.954354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.954272    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:08:40.955249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.955206    5041 cpu_manager.go:414] "containerMap: removing container" podUID="019cadb2-8af5-43b7-8854-bfcb47ef571f" containerName="revoke-keys"
Nov 01 20:08:40.955249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.955227    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019cadb2-8af5-43b7-8854-bfcb47ef571f" containerName="revoke-keys"
Nov 01 20:08:40.955419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.955408    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="019cadb2-8af5-43b7-8854-bfcb47ef571f" containerName="revoke-keys"
Nov 01 20:08:40.959623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.959602    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:40.986911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.986864    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:08:40.992275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.992244    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"]
Nov 01 20:08:40.997237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:40.997210    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.003168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.003143    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:41.003340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.003248    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:41.012689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.012662    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"]
Nov 01 20:08:41.052797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.052750    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:41.053001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.052889    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:41.083594 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083556    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.083716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083655    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.083761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083719    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.083827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083811    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65mg\" (UniqueName: \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.083900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.083937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083917    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.083972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083934    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.083972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083953    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.084030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.083976    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.084030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.084016    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.084094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.084049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpm6\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.084151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.084138    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.184955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.184913    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.184980    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.185135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185056    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65mg\" (UniqueName: \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.185135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185098    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185123    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.185453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185249    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185299    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185344    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185437    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185448    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185461    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpm6\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185544    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185641    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185758    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.185915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185891    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.186032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.185953    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.186097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.186047    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.186387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.186368    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.187474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.187448    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.187651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.187633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.188204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.188179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.196227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.196204    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.196381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.196207    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpm6\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.196816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.196794    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65mg\" (UniqueName: \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\") pod \"maas-api-revoke-sub-7c8fb994-77wzp\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.295702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.295667    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:41.325074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.325050    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:41.609616 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:41.609584    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee4d00a_54c1_4315_a77b_9a81de063dae.slice/crio-62881c9c6db1828370ad477980996a3ecdab3fe8f66c9d54ce67541008477bf8 WatchSource:0}: Error finding container 62881c9c6db1828370ad477980996a3ecdab3fe8f66c9d54ce67541008477bf8: Status 404 returned error can't find the container with id 62881c9c6db1828370ad477980996a3ecdab3fe8f66c9d54ce67541008477bf8
Nov 01 20:08:41.609896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.609866    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:08:41.612132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.612102    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:41.612206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.612167    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:41.612206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.612199    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:41.623285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.623263    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"]
Nov 01 20:08:41.624007 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:41.623985    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7f63ed_4f98_43e9_9fac_728c621d723a.slice/crio-30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad WatchSource:0}: Error finding container 30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad: Status 404 returned error can't find the container with id 30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad
Nov 01 20:08:41.849834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.849767    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp" event={"ID":"cf7f63ed-4f98-43e9-9fac-728c621d723a","Type":"ContainerStarted","Data":"30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad"}
Nov 01 20:08:41.852233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.851698    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" event={"ID":"8ee4d00a-54c1-4315-a77b-9a81de063dae","Type":"ContainerStarted","Data":"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"}
Nov 01 20:08:41.852233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.851787    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" event={"ID":"8ee4d00a-54c1-4315-a77b-9a81de063dae","Type":"ContainerStarted","Data":"62881c9c6db1828370ad477980996a3ecdab3fe8f66c9d54ce67541008477bf8"}
Nov 01 20:08:41.873154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:41.873071    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podStartSLOduration=1.873056703 podStartE2EDuration="1.873056703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:41.869548973 +0000 UTC m=+609.590627650" watchObservedRunningTime="2026-11-01 20:08:41.873056703 +0000 UTC m=+609.594135378"
Nov 01 20:08:42.296445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.296416    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:42.297860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.297832    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:42.298003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.297882    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:42.814975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.814887    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"]
Nov 01 20:08:42.819932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.819903    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:42.833086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.825275    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"]
Nov 01 20:08:42.882509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.882480    5041 generic.go:356] "Generic (PLEG): container finished" podID="cf7f63ed-4f98-43e9-9fac-728c621d723a" containerID="c36f78a7f350ba3b3e6f750996a01db99534c29a3553300205f75a61fb4deefc" exitCode=0
Nov 01 20:08:42.883652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.883618    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp" event={"ID":"cf7f63ed-4f98-43e9-9fac-728c621d723a","Type":"ContainerDied","Data":"c36f78a7f350ba3b3e6f750996a01db99534c29a3553300205f75a61fb4deefc"}
Nov 01 20:08:42.914400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.914363    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:42.914400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.914408    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8kn\" (UniqueName: \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:42.914645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:42.914508    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.016191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.016155    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.016191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.016194    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8kn\" (UniqueName: \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.016410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.016231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.017268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.017243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.018818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.018793    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.024290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.024246    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8kn\" (UniqueName: \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\") pod \"maas-api-revoke-sub-89edd9fb-5mnxz\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.136604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.136522    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:43.296745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.296708    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:43.297210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.296810    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:43.471004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.470651    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"]
Nov 01 20:08:43.475537 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:43.475506    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f253dd7_9dc3_416b_a14f_730c2f5fc993.slice/crio-41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861 WatchSource:0}: Error finding container 41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861: Status 404 returned error can't find the container with id 41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861
Nov 01 20:08:43.892071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.892035    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz" event={"ID":"9f253dd7-9dc3-416b-a14f-730c2f5fc993","Type":"ContainerStarted","Data":"a7e6546c7b7e39051ca068c8e6667155fb14245944bc0a170f263826fc95e32d"}
Nov 01 20:08:43.892230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.892099    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz" event={"ID":"9f253dd7-9dc3-416b-a14f-730c2f5fc993","Type":"ContainerStarted","Data":"41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861"}
Nov 01 20:08:43.907854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:43.907772    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz" podStartSLOduration=1.90775855 podStartE2EDuration="1.90775855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:43.906296777 +0000 UTC m=+611.627375451" watchObservedRunningTime="2026-11-01 20:08:43.90775855 +0000 UTC m=+611.628837226"
Nov 01 20:08:44.085984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.085959    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:44.230280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.230189    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\") pod \"cf7f63ed-4f98-43e9-9fac-728c621d723a\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") "
Nov 01 20:08:44.230439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.230367    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\" (UniqueName: \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\") pod \"cf7f63ed-4f98-43e9-9fac-728c621d723a\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") "
Nov 01 20:08:44.230439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.230423    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\") pod \"cf7f63ed-4f98-43e9-9fac-728c621d723a\" (UID: \"cf7f63ed-4f98-43e9-9fac-728c621d723a\") "
Nov 01 20:08:44.231274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.231238    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca" pod "cf7f63ed-4f98-43e9-9fac-728c621d723a" (UID: "cf7f63ed-4f98-43e9-9fac-728c621d723a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:44.232742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.232720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca" pod "cf7f63ed-4f98-43e9-9fac-728c621d723a" (UID: "cf7f63ed-4f98-43e9-9fac-728c621d723a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:44.232914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.232893    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg" pod "cf7f63ed-4f98-43e9-9fac-728c621d723a" (UID: "cf7f63ed-4f98-43e9-9fac-728c621d723a"). InnerVolumeSpecName "kube-api-access-k65mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:44.297503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.297468    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:44.297880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.297531    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:44.331923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.331895    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:44.331923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.331921    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7f63ed-4f98-43e9-9fac-728c621d723a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:44.332114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.331931    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65mg\" (UniqueName: \"kubernetes.io/projected/cf7f63ed-4f98-43e9-9fac-728c621d723a-kube-api-access-k65mg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:44.940611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.940570    5041 generic.go:356] "Generic (PLEG): container finished" podID="9f253dd7-9dc3-416b-a14f-730c2f5fc993" containerID="a7e6546c7b7e39051ca068c8e6667155fb14245944bc0a170f263826fc95e32d" exitCode=0
Nov 01 20:08:44.941016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.940664    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz" event={"ID":"9f253dd7-9dc3-416b-a14f-730c2f5fc993","Type":"ContainerDied","Data":"a7e6546c7b7e39051ca068c8e6667155fb14245944bc0a170f263826fc95e32d"}
Nov 01 20:08:44.977818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.974220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp" event={"ID":"cf7f63ed-4f98-43e9-9fac-728c621d723a","Type":"ContainerDied","Data":"30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad"}
Nov 01 20:08:44.977818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.974295    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30a5f4679a1a24bd06d998173f5b539bec6e92d123c1be21ecb879b5ff6c5cad"
Nov 01 20:08:44.977818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:44.974383    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"
Nov 01 20:08:45.323808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:45.317464    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:45.323808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:45.317529    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:46.300311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.300276    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:46.300311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.300337    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:46.324603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.324572    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:46.426669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.426640    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\") pod \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") "
Nov 01 20:08:46.426893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.426732    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\") pod \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") "
Nov 01 20:08:46.426967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.426950    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\" (UniqueName: \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\") pod \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\" (UID: \"9f253dd7-9dc3-416b-a14f-730c2f5fc993\") "
Nov 01 20:08:46.427153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.427123    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca" pod "9f253dd7-9dc3-416b-a14f-730c2f5fc993" (UID: "9f253dd7-9dc3-416b-a14f-730c2f5fc993"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:46.427514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.427491    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:46.428917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.428893    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca" pod "9f253dd7-9dc3-416b-a14f-730c2f5fc993" (UID: "9f253dd7-9dc3-416b-a14f-730c2f5fc993"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:46.429562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.429543    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn" pod "9f253dd7-9dc3-416b-a14f-730c2f5fc993" (UID: "9f253dd7-9dc3-416b-a14f-730c2f5fc993"). InnerVolumeSpecName "kube-api-access-dz8kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:46.529249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.529214    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8kn\" (UniqueName: \"kubernetes.io/projected/9f253dd7-9dc3-416b-a14f-730c2f5fc993-kube-api-access-dz8kn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:46.529249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:46.529249    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f253dd7-9dc3-416b-a14f-730c2f5fc993-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:47.000511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.000483    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"
Nov 01 20:08:47.000675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.000519    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz" event={"ID":"9f253dd7-9dc3-416b-a14f-730c2f5fc993","Type":"ContainerDied","Data":"41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861"}
Nov 01 20:08:47.000675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.000568    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41a12c5d097996401e44f946403269f4f05a95d763a275188a5ab255ca0ea861"
Nov 01 20:08:47.296518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.296483    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:47.296708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.296561    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:47.312326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.312296    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:08:47.313131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313108    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cf7f63ed-4f98-43e9-9fac-728c621d723a" containerName="revoke-keys"
Nov 01 20:08:47.313131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313128    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7f63ed-4f98-43e9-9fac-728c621d723a" containerName="revoke-keys"
Nov 01 20:08:47.313333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313180    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9f253dd7-9dc3-416b-a14f-730c2f5fc993" containerName="revoke-keys"
Nov 01 20:08:47.313333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313188    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f253dd7-9dc3-416b-a14f-730c2f5fc993" containerName="revoke-keys"
Nov 01 20:08:47.313333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313317    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f253dd7-9dc3-416b-a14f-730c2f5fc993" containerName="revoke-keys"
Nov 01 20:08:47.313333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.313334    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7f63ed-4f98-43e9-9fac-728c621d723a" containerName="revoke-keys"
Nov 01 20:08:47.317490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.317463    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.321602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.321575    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e38ef4-openshift-default-dockercfg-xpxsg\""
Nov 01 20:08:47.340458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.340432    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441481    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441553    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441582    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441614    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441695    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9t4l\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441745    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.441865    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.443802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.442026    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560668    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560764    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560831    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560857    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560895    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.560972    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.561001    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9t4l\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.561059    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.561152    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.562758    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.563391    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.563638    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.566898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.563955    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.568292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.568267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.569361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.569334    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.572841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.572819    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9t4l\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.572979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.572964    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.574067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.574043    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\") pod \"e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") " pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.640583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.640546    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:47.790499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.790392    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"]
Nov 01 20:08:47.794903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.794870    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.799130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.799105    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:47.799419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.799402    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:47.803665 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.803640    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"]
Nov 01 20:08:47.823941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.823862    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:47.824074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.823986    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:47.869378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.869349    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.869547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.869401    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.869547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.869516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4x56\" (UniqueName: \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.949644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.949618    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:08:47.951765 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:47.951736    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7393207_eea7_4274_ae6e_edb2e1873ad3.slice/crio-7e3b6aa252b9419467aa374c4aae18010c11a5e9c66f15e348946b0e80d77436 WatchSource:0}: Error finding container 7e3b6aa252b9419467aa374c4aae18010c11a5e9c66f15e348946b0e80d77436: Status 404 returned error can't find the container with id 7e3b6aa252b9419467aa374c4aae18010c11a5e9c66f15e348946b0e80d77436
Nov 01 20:08:47.954405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.954373    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:47.954482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.954436    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:47.954482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.954466    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:08:47.970927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.970903    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.971029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.970943    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.971029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.970990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4x56\" (UniqueName: \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.971694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.971671    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.973336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.973308    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:47.979155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:47.979130    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4x56\" (UniqueName: \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\") pod \"maas-api-revoke-sub-0b9bb640-jv8j4\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:48.008420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:48.008395    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerStarted","Data":"7e3b6aa252b9419467aa374c4aae18010c11a5e9c66f15e348946b0e80d77436"}
Nov 01 20:08:48.117159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:48.117071    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:48.296592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:48.296550    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 01 20:08:48.296764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:48.296621    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 01 20:08:48.404050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:48.404025    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"]
Nov 01 20:08:48.405563 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:48.405538    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b4899d_9a54_480e_ad2f_55f7714adb25.slice/crio-b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d WatchSource:0}: Error finding container b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d: Status 404 returned error can't find the container with id b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d
Nov 01 20:08:49.018668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.018565    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerStarted","Data":"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"}
Nov 01 20:08:49.020411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.020384    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4" event={"ID":"96b4899d-9a54-480e-ad2f-55f7714adb25","Type":"ContainerStarted","Data":"6d863433694a2b79f535e65817d5fbd518b6d90954245d5bffc888a9d971cf1b"}
Nov 01 20:08:49.020527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.020431    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4" event={"ID":"96b4899d-9a54-480e-ad2f-55f7714adb25","Type":"ContainerStarted","Data":"b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d"}
Nov 01 20:08:49.042516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.042458    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podStartSLOduration=2.042445536 podStartE2EDuration="2.042445536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:49.040972956 +0000 UTC m=+616.762051634" watchObservedRunningTime="2026-11-01 20:08:49.042445536 +0000 UTC m=+616.763524210"
Nov 01 20:08:49.058990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.058946    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4" podStartSLOduration=2.058931694 podStartE2EDuration="2.058931694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:49.056382742 +0000 UTC m=+616.777461420" watchObservedRunningTime="2026-11-01 20:08:49.058931694 +0000 UTC m=+616.780010370"
Nov 01 20:08:49.641896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.641765    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:49.651285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:49.651249    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:50.029553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.029460    5041 generic.go:356] "Generic (PLEG): container finished" podID="96b4899d-9a54-480e-ad2f-55f7714adb25" containerID="6d863433694a2b79f535e65817d5fbd518b6d90954245d5bffc888a9d971cf1b" exitCode=0
Nov 01 20:08:50.029707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.029548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4" event={"ID":"96b4899d-9a54-480e-ad2f-55f7714adb25","Type":"ContainerDied","Data":"6d863433694a2b79f535e65817d5fbd518b6d90954245d5bffc888a9d971cf1b"}
Nov 01 20:08:50.030011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.029994    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:50.030900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.030881    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:08:50.337973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.337931    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:08:50.338151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.337998    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:08:50.412908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.412878    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:50.413211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.413183    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:50.414205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:50.414181    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:08:51.194192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.194166    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:51.314250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.314164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\") pod \"96b4899d-9a54-480e-ad2f-55f7714adb25\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") "
Nov 01 20:08:51.314419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.314332    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\" (UniqueName: \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\") pod \"96b4899d-9a54-480e-ad2f-55f7714adb25\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") "
Nov 01 20:08:51.314503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.314479    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\") pod \"96b4899d-9a54-480e-ad2f-55f7714adb25\" (UID: \"96b4899d-9a54-480e-ad2f-55f7714adb25\") "
Nov 01 20:08:51.314882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.314842    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca" pod "96b4899d-9a54-480e-ad2f-55f7714adb25" (UID: "96b4899d-9a54-480e-ad2f-55f7714adb25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:51.316394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.316366    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca" pod "96b4899d-9a54-480e-ad2f-55f7714adb25" (UID: "96b4899d-9a54-480e-ad2f-55f7714adb25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:51.316480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.316433    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56" pod "96b4899d-9a54-480e-ad2f-55f7714adb25" (UID: "96b4899d-9a54-480e-ad2f-55f7714adb25"). InnerVolumeSpecName "kube-api-access-b4x56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:51.416573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.416540    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4x56\" (UniqueName: \"kubernetes.io/projected/96b4899d-9a54-480e-ad2f-55f7714adb25-kube-api-access-b4x56\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:51.416573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.416569    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:51.416573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:51.416579    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b4899d-9a54-480e-ad2f-55f7714adb25-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:52.045263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.045216    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4" event={"ID":"96b4899d-9a54-480e-ad2f-55f7714adb25","Type":"ContainerDied","Data":"b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d"}
Nov 01 20:08:52.045263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.045245    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"
Nov 01 20:08:52.045472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.045282    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3681e0087dcbb9f7cfb5d185f9733377e554b780554ef757dc831653df7a37d"
Nov 01 20:08:52.990054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.990021    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"]
Nov 01 20:08:52.990608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.990595    5041 cpu_manager.go:414] "containerMap: removing container" podUID="96b4899d-9a54-480e-ad2f-55f7714adb25" containerName="revoke-keys"
Nov 01 20:08:52.990651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.990609    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b4899d-9a54-480e-ad2f-55f7714adb25" containerName="revoke-keys"
Nov 01 20:08:52.990701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.990692    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b4899d-9a54-480e-ad2f-55f7714adb25" containerName="revoke-keys"
Nov 01 20:08:52.994447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.994432    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:52.999171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.999143    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:52.999311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:52.999258    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:53.003023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.003000    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"]
Nov 01 20:08:53.032021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.031992    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"]
Nov 01 20:08:53.035987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.035971    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.044600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.044577    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"]
Nov 01 20:08:53.076063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.076028    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:53.076660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.076644    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:53.155894 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.155858    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.156045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.155964    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47r2r\" (UniqueName: \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.156045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.155993    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.156145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.156086    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.156193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.156180    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.156278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.156266    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcn8\" (UniqueName: \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.257834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.257733    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.257998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.257833    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47r2r\" (UniqueName: \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.257998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.257873    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.257998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.257909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.257998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.257964    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.258223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.258038    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcn8\" (UniqueName: \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.258597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.258560    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.258716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.258635    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.260325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.260301    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.260410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.260340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.265385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.265362    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcn8\" (UniqueName: \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\") pod \"maas-api-revoke-sub-6acf3a5a-2s62j\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.265385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.265378    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47r2r\" (UniqueName: \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\") pod \"maas-api-revoke-sub-05deae6b-bw6cb\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.311229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.311201    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:53.352064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.352030    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:53.595991 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:53.595956    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc328c37_b985_4099_8ca8_19b753a53b3a.slice/crio-794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97 WatchSource:0}: Error finding container 794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97: Status 404 returned error can't find the container with id 794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97
Nov 01 20:08:53.596117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.596089    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"]
Nov 01 20:08:53.645126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:53.645095    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"]
Nov 01 20:08:53.645550 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:53.645528    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e29abde_1803_404c_ad0e_d7c8884f8946.slice/crio-1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d WatchSource:0}: Error finding container 1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d: Status 404 returned error can't find the container with id 1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d
Nov 01 20:08:54.061475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.061439    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j" event={"ID":"bc328c37-b985-4099-8ca8-19b753a53b3a","Type":"ContainerStarted","Data":"067aefd68c69c3a10d7157a6d902da3c59bb7faa977c1001d7eeeabfed6c9ba6"}
Nov 01 20:08:54.061957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.061507    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j" event={"ID":"bc328c37-b985-4099-8ca8-19b753a53b3a","Type":"ContainerStarted","Data":"794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97"}
Nov 01 20:08:54.063088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.063057    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb" event={"ID":"9e29abde-1803-404c-ad0e-d7c8884f8946","Type":"ContainerStarted","Data":"76ae381f976282ce1078ab2f3505b6a350cda2d88b17e0d70c1539c8745301c2"}
Nov 01 20:08:54.063203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.063108    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb" event={"ID":"9e29abde-1803-404c-ad0e-d7c8884f8946","Type":"ContainerStarted","Data":"1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d"}
Nov 01 20:08:54.080749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.080698    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j" podStartSLOduration=2.080684735 podStartE2EDuration="2.080684735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:54.076196644 +0000 UTC m=+621.797275323" watchObservedRunningTime="2026-11-01 20:08:54.080684735 +0000 UTC m=+621.801763410"
Nov 01 20:08:54.092117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:54.092074    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb" podStartSLOduration=1.092063714 podStartE2EDuration="1.092063714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:54.090099975 +0000 UTC m=+621.811178652" watchObservedRunningTime="2026-11-01 20:08:54.092063714 +0000 UTC m=+621.813142390"
Nov 01 20:08:55.081830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:55.080733    5041 generic.go:356] "Generic (PLEG): container finished" podID="bc328c37-b985-4099-8ca8-19b753a53b3a" containerID="067aefd68c69c3a10d7157a6d902da3c59bb7faa977c1001d7eeeabfed6c9ba6" exitCode=0
Nov 01 20:08:55.081830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:55.080906    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j" event={"ID":"bc328c37-b985-4099-8ca8-19b753a53b3a","Type":"ContainerDied","Data":"067aefd68c69c3a10d7157a6d902da3c59bb7faa977c1001d7eeeabfed6c9ba6"}
Nov 01 20:08:55.085265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:55.085239    5041 generic.go:356] "Generic (PLEG): container finished" podID="9e29abde-1803-404c-ad0e-d7c8884f8946" containerID="76ae381f976282ce1078ab2f3505b6a350cda2d88b17e0d70c1539c8745301c2" exitCode=0
Nov 01 20:08:55.085656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:55.085535    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb" event={"ID":"9e29abde-1803-404c-ad0e-d7c8884f8946","Type":"ContainerDied","Data":"76ae381f976282ce1078ab2f3505b6a350cda2d88b17e0d70c1539c8745301c2"}
Nov 01 20:08:56.433583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.433552    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:56.478254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.478228    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:56.532419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.532383    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\") pod \"bc328c37-b985-4099-8ca8-19b753a53b3a\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") "
Nov 01 20:08:56.532660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.532459    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\") pod \"bc328c37-b985-4099-8ca8-19b753a53b3a\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") "
Nov 01 20:08:56.532660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.532489    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\" (UniqueName: \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\") pod \"bc328c37-b985-4099-8ca8-19b753a53b3a\" (UID: \"bc328c37-b985-4099-8ca8-19b753a53b3a\") "
Nov 01 20:08:56.532873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.532844    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca" pod "bc328c37-b985-4099-8ca8-19b753a53b3a" (UID: "bc328c37-b985-4099-8ca8-19b753a53b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:56.533282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.533078    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:56.534723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.534702    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8" pod "bc328c37-b985-4099-8ca8-19b753a53b3a" (UID: "bc328c37-b985-4099-8ca8-19b753a53b3a"). InnerVolumeSpecName "kube-api-access-dlcn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:56.535131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.535115    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca" pod "bc328c37-b985-4099-8ca8-19b753a53b3a" (UID: "bc328c37-b985-4099-8ca8-19b753a53b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:56.633753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.633722    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\") pod \"9e29abde-1803-404c-ad0e-d7c8884f8946\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") "
Nov 01 20:08:56.633928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.633798    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\") pod \"9e29abde-1803-404c-ad0e-d7c8884f8946\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") "
Nov 01 20:08:56.633928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.633922    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\" (UniqueName: \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\") pod \"9e29abde-1803-404c-ad0e-d7c8884f8946\" (UID: \"9e29abde-1803-404c-ad0e-d7c8884f8946\") "
Nov 01 20:08:56.634234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.634212    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc328c37-b985-4099-8ca8-19b753a53b3a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:56.634234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.634230    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcn8\" (UniqueName: \"kubernetes.io/projected/bc328c37-b985-4099-8ca8-19b753a53b3a-kube-api-access-dlcn8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:56.634379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.634238    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca" pod "9e29abde-1803-404c-ad0e-d7c8884f8946" (UID: "9e29abde-1803-404c-ad0e-d7c8884f8946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:08:56.635909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.635886    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca" pod "9e29abde-1803-404c-ad0e-d7c8884f8946" (UID: "9e29abde-1803-404c-ad0e-d7c8884f8946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:08:56.635980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.635958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r" pod "9e29abde-1803-404c-ad0e-d7c8884f8946" (UID: "9e29abde-1803-404c-ad0e-d7c8884f8946"). InnerVolumeSpecName "kube-api-access-47r2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:08:56.735815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.735681    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47r2r\" (UniqueName: \"kubernetes.io/projected/9e29abde-1803-404c-ad0e-d7c8884f8946-kube-api-access-47r2r\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:56.735815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.735716    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:56.735815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:56.735727    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e29abde-1803-404c-ad0e-d7c8884f8946-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:08:57.106839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.106803    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb" event={"ID":"9e29abde-1803-404c-ad0e-d7c8884f8946","Type":"ContainerDied","Data":"1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d"}
Nov 01 20:08:57.107031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.106859    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1983eccde76c20f2a385ccf9685a5317933968cbdc53bfe9f80cbbf2e6da0f3d"
Nov 01 20:08:57.107031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.106813    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"
Nov 01 20:08:57.108569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.108537    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j" event={"ID":"bc328c37-b985-4099-8ca8-19b753a53b3a","Type":"ContainerDied","Data":"794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97"}
Nov 01 20:08:57.108569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.108562    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"
Nov 01 20:08:57.108717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.108580    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794e2555e65b25b1f8a6c3121f738fb285611efd5144d3e070f9537cb5e32c97"
Nov 01 20:08:57.978944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.978906    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"]
Nov 01 20:08:57.979653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979632    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc328c37-b985-4099-8ca8-19b753a53b3a" containerName="revoke-keys"
Nov 01 20:08:57.979754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979656    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc328c37-b985-4099-8ca8-19b753a53b3a" containerName="revoke-keys"
Nov 01 20:08:57.979754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979673    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9e29abde-1803-404c-ad0e-d7c8884f8946" containerName="revoke-keys"
Nov 01 20:08:57.979754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979680    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e29abde-1803-404c-ad0e-d7c8884f8946" containerName="revoke-keys"
Nov 01 20:08:57.979945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979916    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc328c37-b985-4099-8ca8-19b753a53b3a" containerName="revoke-keys"
Nov 01 20:08:57.979945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.979932    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e29abde-1803-404c-ad0e-d7c8884f8946" containerName="revoke-keys"
Nov 01 20:08:57.984882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.984859    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:57.988655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.988632    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:08:57.990165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.989458    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:08:57.990165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.989494    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:08:57.990165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.989494    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:08:57.990165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:57.989545    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"]
Nov 01 20:08:58.051511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.051474    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxsmd\" (UniqueName: \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.051511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.051511    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.051719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.051545    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.153219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.153184    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxsmd\" (UniqueName: \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.153219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.153223    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.153432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.153255    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.153934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.153911    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.155745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.155721    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.160732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.160707    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxsmd\" (UniqueName: \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\") pod \"maas-api-revoke-sub-8854ed13-ffdbw\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.301845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.301815    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:08:58.541238 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:08:58.541202    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:08:58.580018 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:08:58.579989    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1869d6_4648_498c_a591_cb2e14fb49b0.slice/crio-794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70 WatchSource:0}: Error finding container 794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70: Status 404 returned error can't find the container with id 794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70
Nov 01 20:08:58.581271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:58.581245    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"]
Nov 01 20:08:59.123399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:59.123305    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw" event={"ID":"5e1869d6-4648-498c-a591-cb2e14fb49b0","Type":"ContainerStarted","Data":"90a07340d3098a0c7e0c0698e6d9933f02ef63c7cf07c401de31064176d3cd5c"}
Nov 01 20:08:59.123399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:59.123369    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw" event={"ID":"5e1869d6-4648-498c-a591-cb2e14fb49b0","Type":"ContainerStarted","Data":"794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70"}
Nov 01 20:08:59.141745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:08:59.141694    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw" podStartSLOduration=2.141680951 podStartE2EDuration="2.141680951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:59.137961149 +0000 UTC m=+626.859039830" watchObservedRunningTime="2026-11-01 20:08:59.141680951 +0000 UTC m=+626.862759626"
Nov 01 20:09:00.131502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:00.131465    5041 generic.go:356] "Generic (PLEG): container finished" podID="5e1869d6-4648-498c-a591-cb2e14fb49b0" containerID="90a07340d3098a0c7e0c0698e6d9933f02ef63c7cf07c401de31064176d3cd5c" exitCode=0
Nov 01 20:09:00.131999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:00.131527    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw" event={"ID":"5e1869d6-4648-498c-a591-cb2e14fb49b0","Type":"ContainerDied","Data":"90a07340d3098a0c7e0c0698e6d9933f02ef63c7cf07c401de31064176d3cd5c"}
Nov 01 20:09:01.299763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.299731    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:09:01.392651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.392561    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\") pod \"5e1869d6-4648-498c-a591-cb2e14fb49b0\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") "
Nov 01 20:09:01.392827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.392737    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\") pod \"5e1869d6-4648-498c-a591-cb2e14fb49b0\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") "
Nov 01 20:09:01.392827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.392808    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\" (UniqueName: \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\") pod \"5e1869d6-4648-498c-a591-cb2e14fb49b0\" (UID: \"5e1869d6-4648-498c-a591-cb2e14fb49b0\") "
Nov 01 20:09:01.392973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.392950    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca" pod "5e1869d6-4648-498c-a591-cb2e14fb49b0" (UID: "5e1869d6-4648-498c-a591-cb2e14fb49b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:01.393277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.393258    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:01.394955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.394929    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd" pod "5e1869d6-4648-498c-a591-cb2e14fb49b0" (UID: "5e1869d6-4648-498c-a591-cb2e14fb49b0"). InnerVolumeSpecName "kube-api-access-mxsmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:01.395058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.394966    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca" pod "5e1869d6-4648-498c-a591-cb2e14fb49b0" (UID: "5e1869d6-4648-498c-a591-cb2e14fb49b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:01.494788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.494748    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1869d6-4648-498c-a591-cb2e14fb49b0-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:01.494957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:01.494813    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxsmd\" (UniqueName: \"kubernetes.io/projected/5e1869d6-4648-498c-a591-cb2e14fb49b0-kube-api-access-mxsmd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:02.146635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:02.146603    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw" event={"ID":"5e1869d6-4648-498c-a591-cb2e14fb49b0","Type":"ContainerDied","Data":"794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70"}
Nov 01 20:09:02.146635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:02.146631    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"
Nov 01 20:09:02.146892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:02.146653    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794836847aa027da4b1d93e7771a4d098520696270aa9e54b1fc0e802a699d70"
Nov 01 20:09:06.592804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.583887    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"]
Nov 01 20:09:06.592804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.592204    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5e1869d6-4648-498c-a591-cb2e14fb49b0" containerName="revoke-keys"
Nov 01 20:09:06.592804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.592229    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1869d6-4648-498c-a591-cb2e14fb49b0" containerName="revoke-keys"
Nov 01 20:09:06.592804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.592382    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1869d6-4648-498c-a591-cb2e14fb49b0" containerName="revoke-keys"
Nov 01 20:09:06.613514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.613485    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"]
Nov 01 20:09:06.614486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.613948    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.624395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.624165    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:06.624899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.624876    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:06.625129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.625113    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:06.625349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.625329    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:06.726127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.726091    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29psn\" (UniqueName: \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.726328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.726303    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.726393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.726357    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.827758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.827720    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.827932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.827773    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.827970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.827933    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29psn\" (UniqueName: \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.828499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.828473    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.830300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.830274    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.835728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.835694    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29psn\" (UniqueName: \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\") pod \"maas-api-revoke-sub-019514d2-5qtxf\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:06.941649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:06.941556    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:07.255084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:07.254979    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"]
Nov 01 20:09:07.274753 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:07.274720    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab8cb35_3d15_43ab_9a7e_a1dc6317d335.slice/crio-264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70 WatchSource:0}: Error finding container 264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70: Status 404 returned error can't find the container with id 264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70
Nov 01 20:09:08.219948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.219910    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf" event={"ID":"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335","Type":"ContainerStarted","Data":"c13094b8c682854f8602b4492d1e3534fc5d4e5934c119f0318e24ae507aaded"}
Nov 01 20:09:08.220461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.219977    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf" event={"ID":"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335","Type":"ContainerStarted","Data":"264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70"}
Nov 01 20:09:08.237983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.237908    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf" podStartSLOduration=2.23789389 podStartE2EDuration="2.23789389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:08.234891086 +0000 UTC m=+635.955969762" watchObservedRunningTime="2026-11-01 20:09:08.23789389 +0000 UTC m=+635.958972578"
Nov 01 20:09:08.623505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.623457    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"]
Nov 01 20:09:08.628481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.628444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"]
Nov 01 20:09:08.628629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.628483    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.779459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.779423    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9fg\" (UniqueName: \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.779459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.779465    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.779667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.779486    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.881370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.881277    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9fg\" (UniqueName: \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.881370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.881326    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.881370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.881362    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.882056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.882036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.884396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.884365    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.889018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.888993    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9fg\" (UniqueName: \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\") pod \"maas-api-revoke-sub-7129d2a5-4c268\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:08.974607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:08.974572    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:09.228145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:09.228050    5041 generic.go:356] "Generic (PLEG): container finished" podID="0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" containerID="c13094b8c682854f8602b4492d1e3534fc5d4e5934c119f0318e24ae507aaded" exitCode=0
Nov 01 20:09:09.228145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:09.228091    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf" event={"ID":"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335","Type":"ContainerDied","Data":"c13094b8c682854f8602b4492d1e3534fc5d4e5934c119f0318e24ae507aaded"}
Nov 01 20:09:09.292537 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:09.292472    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd0c044_8ad7_42a2_9275_fd7c412f5716.slice/crio-6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd WatchSource:0}: Error finding container 6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd: Status 404 returned error can't find the container with id 6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd
Nov 01 20:09:09.308729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:09.308697    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"]
Nov 01 20:09:10.241772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.241733    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268" event={"ID":"0cd0c044-8ad7-42a2-9275-fd7c412f5716","Type":"ContainerStarted","Data":"1b200883ac34cd5b79bfd908017642e2186107c7ab7192a8ca95d16aff9ddda7"}
Nov 01 20:09:10.242294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.241847    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268" event={"ID":"0cd0c044-8ad7-42a2-9275-fd7c412f5716","Type":"ContainerStarted","Data":"6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd"}
Nov 01 20:09:10.283233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.283181    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268" podStartSLOduration=2.283164195 podStartE2EDuration="2.283164195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:10.280957779 +0000 UTC m=+638.002036456" watchObservedRunningTime="2026-11-01 20:09:10.283164195 +0000 UTC m=+638.004242913"
Nov 01 20:09:10.397609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.397578    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"]
Nov 01 20:09:10.402407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.402382    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.410559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.410364    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"]
Nov 01 20:09:10.508481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.508423    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:10.512305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.512277    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltclf\" (UniqueName: \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.512421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.512339    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.512628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.512608    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.613344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613298    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\") pod \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") "
Nov 01 20:09:10.613344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613352    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\" (UniqueName: \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\") pod \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") "
Nov 01 20:09:10.613556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613494    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\") pod \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\" (UID: \"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335\") "
Nov 01 20:09:10.613629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613599    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltclf\" (UniqueName: \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.613667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613635    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.613703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613669    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.613794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613742    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca" pod "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" (UID: "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:10.613961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.613936    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:10.614431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.614403    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.615673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.615651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn" pod "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" (UID: "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335"). InnerVolumeSpecName "kube-api-access-29psn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:10.615768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.615689    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca" pod "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" (UID: "0ab8cb35-3d15-43ab-9a7e-a1dc6317d335"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:10.616010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.615992    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.624834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.624813    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltclf\" (UniqueName: \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\") pod \"maas-api-revoke-sub-7f77b5ab-9fbbk\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:10.715383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.715349    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:10.715383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.715378    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29psn\" (UniqueName: \"kubernetes.io/projected/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335-kube-api-access-29psn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:10.758104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:10.741964    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:11.082827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.082796    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"]
Nov 01 20:09:11.083259 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:11.083225    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e61d48_ccb9_435a_8970_8e9a7b611aa4.slice/crio-f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f WatchSource:0}: Error finding container f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f: Status 404 returned error can't find the container with id f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f
Nov 01 20:09:11.115810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.115768    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:09:11.116956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.116586    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" containerName="revoke-keys"
Nov 01 20:09:11.116956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.116609    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" containerName="revoke-keys"
Nov 01 20:09:11.116956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.116814    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" containerName="revoke-keys"
Nov 01 20:09:11.121849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.121826    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.131211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.131182    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:09:11.207683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.207650    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:09:11.211758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.211735    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.221573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.221549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wmd\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.221789 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.221757    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.221943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.221927    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222086    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222317    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222456    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222599    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.222733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222710    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:09:11.222825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.222718    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.294852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.294811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk" event={"ID":"57e61d48-ccb9-435a-8970-8e9a7b611aa4","Type":"ContainerStarted","Data":"f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f"}
Nov 01 20:09:11.296437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.296407    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf" event={"ID":"0ab8cb35-3d15-43ab-9a7e-a1dc6317d335","Type":"ContainerDied","Data":"264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70"}
Nov 01 20:09:11.296556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.296451    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="264605dd55a08d848a602e55ee7f45a0919d2946aa2c110f7f0b5405f395fe70"
Nov 01 20:09:11.296556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.296506    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"
Nov 01 20:09:11.302295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.302269    5041 generic.go:356] "Generic (PLEG): container finished" podID="0cd0c044-8ad7-42a2-9275-fd7c412f5716" containerID="1b200883ac34cd5b79bfd908017642e2186107c7ab7192a8ca95d16aff9ddda7" exitCode=0
Nov 01 20:09:11.302379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.302310    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268" event={"ID":"0cd0c044-8ad7-42a2-9275-fd7c412f5716","Type":"ContainerDied","Data":"1b200883ac34cd5b79bfd908017642e2186107c7ab7192a8ca95d16aff9ddda7"}
Nov 01 20:09:11.323763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323737    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323793    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323824    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323854    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323876    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323893    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323911    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxs9\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323930    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323949    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.323965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323968    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.324246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.323984    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.324246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324000    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wmd\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.324246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324020    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.324246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.324365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324341    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.324451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324429    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.324812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324470    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.325011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.325142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.325128    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.325241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.325229    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.325360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.324696    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.325432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.325351    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.325993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.325969    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.326609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.326591    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.329202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.329178    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.333204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.333161    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wmd\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.333424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.333411    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.426685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426647    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426685    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426710    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxs9\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426735    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426769    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426819    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426850    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.426946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426928    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.427241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.426986    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.427471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.427451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.427856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.427835    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.428249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.428228    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.428564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.428543    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.428639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.428556    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.429909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.429890    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.430381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.430360    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.436421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.436399    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxs9\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.436536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.436465    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.479323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.479290    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:11.528140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.528103    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:11.762965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.762940    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:09:11.763353 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:11.763325    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1940b5aa_9298_4837_bff7_32abb2bda4b8.slice/crio-8674ea658c6b8591370a71177c473242fa70a613fe4815272c5eaaa9b550d88e WatchSource:0}: Error finding container 8674ea658c6b8591370a71177c473242fa70a613fe4815272c5eaaa9b550d88e: Status 404 returned error can't find the container with id 8674ea658c6b8591370a71177c473242fa70a613fe4815272c5eaaa9b550d88e
Nov 01 20:09:11.765773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.765736    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:11.765873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.765842    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:11.765920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.765878    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:11.847688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.847616    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:09:11.853023 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:11.852998    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2270b821_22ea_4bb2_bc5e_7bcea6468704.slice/crio-33ae15e1b0543e02a43ae888c9487f3e5c8f99c1ef0ec55d4a2778b2dfadcc13 WatchSource:0}: Error finding container 33ae15e1b0543e02a43ae888c9487f3e5c8f99c1ef0ec55d4a2778b2dfadcc13: Status 404 returned error can't find the container with id 33ae15e1b0543e02a43ae888c9487f3e5c8f99c1ef0ec55d4a2778b2dfadcc13
Nov 01 20:09:11.856264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.856212    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:11.856351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.856333    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:11.856406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:11.856381    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:12.313726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.313690    5041 generic.go:356] "Generic (PLEG): container finished" podID="57e61d48-ccb9-435a-8970-8e9a7b611aa4" containerID="ae5926f5457aaaa58bd9dc00aebbf76823903b0044e3d90b2a50b8c33eec582c" exitCode=0
Nov 01 20:09:12.314243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.313797    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk" event={"ID":"57e61d48-ccb9-435a-8970-8e9a7b611aa4","Type":"ContainerDied","Data":"ae5926f5457aaaa58bd9dc00aebbf76823903b0044e3d90b2a50b8c33eec582c"}
Nov 01 20:09:12.315929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.315892    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" event={"ID":"1940b5aa-9298-4837-bff7-32abb2bda4b8","Type":"ContainerStarted","Data":"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"}
Nov 01 20:09:12.316052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.315947    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" event={"ID":"1940b5aa-9298-4837-bff7-32abb2bda4b8","Type":"ContainerStarted","Data":"8674ea658c6b8591370a71177c473242fa70a613fe4815272c5eaaa9b550d88e"}
Nov 01 20:09:12.320548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.320520    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" event={"ID":"2270b821-22ea-4bb2-bc5e-7bcea6468704","Type":"ContainerStarted","Data":"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"}
Nov 01 20:09:12.320703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.320574    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" event={"ID":"2270b821-22ea-4bb2-bc5e-7bcea6468704","Type":"ContainerStarted","Data":"33ae15e1b0543e02a43ae888c9487f3e5c8f99c1ef0ec55d4a2778b2dfadcc13"}
Nov 01 20:09:12.359387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.359267    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podStartSLOduration=1.359245082 podStartE2EDuration="1.359245082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:12.356438397 +0000 UTC m=+640.077517076" watchObservedRunningTime="2026-11-01 20:09:12.359245082 +0000 UTC m=+640.080323811"
Nov 01 20:09:12.414928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.414792    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podStartSLOduration=1.414743 podStartE2EDuration="1.414743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:12.406550991 +0000 UTC m=+640.127629668" watchObservedRunningTime="2026-11-01 20:09:12.414743 +0000 UTC m=+640.135821685"
Nov 01 20:09:12.497803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.480205    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:12.513150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.510555    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:12.513150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.510622    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:12.598146 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.549882    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:12.679318 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.679164    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:12.719433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.719398    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\" (UniqueName: \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\") pod \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") "
Nov 01 20:09:12.719729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.719707    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\") pod \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") "
Nov 01 20:09:12.719914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.719896    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\") pod \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\" (UID: \"0cd0c044-8ad7-42a2-9275-fd7c412f5716\") "
Nov 01 20:09:12.720903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.720878    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca" pod "0cd0c044-8ad7-42a2-9275-fd7c412f5716" (UID: "0cd0c044-8ad7-42a2-9275-fd7c412f5716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:12.723628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.723591    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca" pod "0cd0c044-8ad7-42a2-9275-fd7c412f5716" (UID: "0cd0c044-8ad7-42a2-9275-fd7c412f5716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:12.723983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.723956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg" pod "0cd0c044-8ad7-42a2-9275-fd7c412f5716" (UID: "0cd0c044-8ad7-42a2-9275-fd7c412f5716"). InnerVolumeSpecName "kube-api-access-xk9fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:12.828199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.827939    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:12.828199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.827976    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0c044-8ad7-42a2-9275-fd7c412f5716-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:12.828199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:12.827995    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9fg\" (UniqueName: \"kubernetes.io/projected/0cd0c044-8ad7-42a2-9275-fd7c412f5716-kube-api-access-xk9fg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:13.509803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.505106    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"
Nov 01 20:09:13.551722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.528866    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268" event={"ID":"0cd0c044-8ad7-42a2-9275-fd7c412f5716","Type":"ContainerDied","Data":"6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd"}
Nov 01 20:09:13.551722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.528957    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea529588dec8a4233430e33536379cd6775dc505dd63af0fd386da4f9c31dcd"
Nov 01 20:09:13.552915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.552403    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:13.552915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.552468    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:13.582072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.582037    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:13.582624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:13.582582    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:14.545807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.523820    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:14.545807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.523879    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:14.632210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.559118    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:14.632210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.559177    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:14.885858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.845011    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:14.981195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:14.967179    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:15.137170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.137019    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:15.312802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.312527    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\") pod \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") "
Nov 01 20:09:15.312802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.312570    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\") pod \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") "
Nov 01 20:09:15.312802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.312762    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\" (UniqueName: \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\") pod \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\" (UID: \"57e61d48-ccb9-435a-8970-8e9a7b611aa4\") "
Nov 01 20:09:15.313337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.313304    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca" pod "57e61d48-ccb9-435a-8970-8e9a7b611aa4" (UID: "57e61d48-ccb9-435a-8970-8e9a7b611aa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:15.313607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.313580    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:15.315483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.315454    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca" pod "57e61d48-ccb9-435a-8970-8e9a7b611aa4" (UID: "57e61d48-ccb9-435a-8970-8e9a7b611aa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:15.315743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.315712    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf" pod "57e61d48-ccb9-435a-8970-8e9a7b611aa4" (UID: "57e61d48-ccb9-435a-8970-8e9a7b611aa4"). InnerVolumeSpecName "kube-api-access-ltclf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:15.414871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.414794    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57e61d48-ccb9-435a-8970-8e9a7b611aa4-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:15.414871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.414827    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltclf\" (UniqueName: \"kubernetes.io/projected/57e61d48-ccb9-435a-8970-8e9a7b611aa4-kube-api-access-ltclf\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:15.487624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.487189    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:15.487624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.487268    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:15.530805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.530362    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:15.530805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.530434    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:15.713562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.713467    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"
Nov 01 20:09:15.713562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.713496    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk" event={"ID":"57e61d48-ccb9-435a-8970-8e9a7b611aa4","Type":"ContainerDied","Data":"f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f"}
Nov 01 20:09:15.713562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:15.713544    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7d500b40f004b218cd19fa7d6eae3e032fdfe653d8553722e28729f4b29177f"
Nov 01 20:09:16.482203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.482162    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:16.482425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.482229    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:16.529651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.529614    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:16.529871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.529661    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:16.811460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.811427    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"]
Nov 01 20:09:16.812068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812053    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0cd0c044-8ad7-42a2-9275-fd7c412f5716" containerName="revoke-keys"
Nov 01 20:09:16.812068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812069    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd0c044-8ad7-42a2-9275-fd7c412f5716" containerName="revoke-keys"
Nov 01 20:09:16.812167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812093    5041 cpu_manager.go:414] "containerMap: removing container" podUID="57e61d48-ccb9-435a-8970-8e9a7b611aa4" containerName="revoke-keys"
Nov 01 20:09:16.812167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812099    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e61d48-ccb9-435a-8970-8e9a7b611aa4" containerName="revoke-keys"
Nov 01 20:09:16.812231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812179    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd0c044-8ad7-42a2-9275-fd7c412f5716" containerName="revoke-keys"
Nov 01 20:09:16.812231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.812191    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e61d48-ccb9-435a-8970-8e9a7b611aa4" containerName="revoke-keys"
Nov 01 20:09:16.815817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.815796    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.820037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.819948    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:16.820037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.819995    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:16.820037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.819977    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:16.820269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.820145    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:16.823610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.823588    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"]
Nov 01 20:09:16.835701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.833399    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.835701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.833518    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzjc\" (UniqueName: \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.835701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.833725    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.936095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.934822    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.936095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.934885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzjc\" (UniqueName: \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.936095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.935046    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.936095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.935881    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.937874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.937849    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:16.981628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:16.981592    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzjc\" (UniqueName: \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\") pod \"maas-api-revoke-sub-f4bd439f-bnxgf\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:17.135438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.135283    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:17.500061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.484028    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:17.500061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.484106    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:17.529724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.529690    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:17.529891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.529743    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:17.739772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:17.739507    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"]
Nov 01 20:09:18.480663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.480183    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 01 20:09:18.480663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.480248    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 01 20:09:18.529806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.529758    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:18.529985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.529832    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:18.738787 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.738677    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf" event={"ID":"86ec6baa-35a6-4538-bfbd-833185672008","Type":"ContainerStarted","Data":"a7d99a56a853152ee965b5f4082093ba888ee6bd223dfdea7717152947449df6"}
Nov 01 20:09:18.738949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.738798    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf" event={"ID":"86ec6baa-35a6-4538-bfbd-833185672008","Type":"ContainerStarted","Data":"ba2f1b985fcaf710807267b0f863eeb9bd916b8a2ff6b7f071322c2b06fcb76e"}
Nov 01 20:09:18.784399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.784311    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf" podStartSLOduration=2.7842907070000003 podStartE2EDuration="2.784290707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:18.775199651 +0000 UTC m=+646.496278329" watchObservedRunningTime="2026-11-01 20:09:18.784290707 +0000 UTC m=+646.505369384"
Nov 01 20:09:18.901988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.901950    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"]
Nov 01 20:09:18.906622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.906601    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:18.919732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.918593    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"]
Nov 01 20:09:18.919732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.919614    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:18.919920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.919761    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jbjz\" (UniqueName: \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:18.919920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:18.919898    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.020865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.020763    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbjz\" (UniqueName: \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.021045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.020918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.021114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.021044    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.021732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.021708    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.023925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.023873    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.028708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.028659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbjz\" (UniqueName: \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\") pod \"maas-api-revoke-sub-2d85f7a5-lqpr9\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.225259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.225218    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:19.531321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.530883    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 01 20:09:19.531321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.530942    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 01 20:09:19.643212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.643109    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"]
Nov 01 20:09:19.643212 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:19.643160    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode272e37a_f4ca_428e_b9c7_f381eb55b913.slice/crio-ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c WatchSource:0}: Error finding container ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c: Status 404 returned error can't find the container with id ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c
Nov 01 20:09:19.752145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.752109    5041 generic.go:356] "Generic (PLEG): container finished" podID="86ec6baa-35a6-4538-bfbd-833185672008" containerID="a7d99a56a853152ee965b5f4082093ba888ee6bd223dfdea7717152947449df6" exitCode=0
Nov 01 20:09:19.752423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.752194    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf" event={"ID":"86ec6baa-35a6-4538-bfbd-833185672008","Type":"ContainerDied","Data":"a7d99a56a853152ee965b5f4082093ba888ee6bd223dfdea7717152947449df6"}
Nov 01 20:09:19.754259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:19.754228    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9" event={"ID":"e272e37a-f4ca-428e-b9c7-f381eb55b913","Type":"ContainerStarted","Data":"ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c"}
Nov 01 20:09:20.489573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.486618    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:38578->10.132.0.92:15021: read: connection reset by peer" start-of-body=
Nov 01 20:09:20.489573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.486744    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": read tcp 10.132.0.2:38578->10.132.0.92:15021: read: connection reset by peer"
Nov 01 20:09:20.523790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.522893    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:09:20.523790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.522963    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:20.539637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.539398    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:20.540547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.540514    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:20.541608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.541582    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:09:20.585350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.585275    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:20.587110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.586589    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:20.587521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.587489    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:09:20.763053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.763026    5041 generic.go:356] "Generic (PLEG): container finished" podID="e272e37a-f4ca-428e-b9c7-f381eb55b913" containerID="68760da68c26fe1fdad15559be409efe13a7b0d71342e67bf87ba616ba0331df" exitCode=0
Nov 01 20:09:20.763167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.763103    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9" event={"ID":"e272e37a-f4ca-428e-b9c7-f381eb55b913","Type":"ContainerDied","Data":"68760da68c26fe1fdad15559be409efe13a7b0d71342e67bf87ba616ba0331df"}
Nov 01 20:09:20.765148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.765125    5041 generic.go:356] "Generic (PLEG): container finished" podID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerID="1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27" exitCode=0
Nov 01 20:09:20.765258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.765212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerDied","Data":"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"}
Nov 01 20:09:20.766109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.766031    5041 scope.go:122] "RemoveContainer" containerID="1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"
Nov 01 20:09:20.767487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.767458    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:20.767591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.767531    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:20.767591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:20.767563    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:21.101716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.101681    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:21.173530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.173034    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\" (UniqueName: \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\") pod \"86ec6baa-35a6-4538-bfbd-833185672008\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") "
Nov 01 20:09:21.175987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.175955    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc" pod "86ec6baa-35a6-4538-bfbd-833185672008" (UID: "86ec6baa-35a6-4538-bfbd-833185672008"). InnerVolumeSpecName "kube-api-access-9rzjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:21.274170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.274138    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\") pod \"86ec6baa-35a6-4538-bfbd-833185672008\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") "
Nov 01 20:09:21.274378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.274213    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\") pod \"86ec6baa-35a6-4538-bfbd-833185672008\" (UID: \"86ec6baa-35a6-4538-bfbd-833185672008\") "
Nov 01 20:09:21.274608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.274587    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzjc\" (UniqueName: \"kubernetes.io/projected/86ec6baa-35a6-4538-bfbd-833185672008-kube-api-access-9rzjc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:21.274680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.274608    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca" pod "86ec6baa-35a6-4538-bfbd-833185672008" (UID: "86ec6baa-35a6-4538-bfbd-833185672008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:21.276516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.276487    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca" pod "86ec6baa-35a6-4538-bfbd-833185672008" (UID: "86ec6baa-35a6-4538-bfbd-833185672008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:21.375565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.375482    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:21.375565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.375522    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ec6baa-35a6-4538-bfbd-833185672008-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:21.781369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.781274    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerStarted","Data":"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"}
Nov 01 20:09:21.782953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.782900    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"
Nov 01 20:09:21.783105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.782942    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf" event={"ID":"86ec6baa-35a6-4538-bfbd-833185672008","Type":"ContainerDied","Data":"ba2f1b985fcaf710807267b0f863eeb9bd916b8a2ff6b7f071322c2b06fcb76e"}
Nov 01 20:09:21.783260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:21.783244    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2f1b985fcaf710807267b0f863eeb9bd916b8a2ff6b7f071322c2b06fcb76e"
Nov 01 20:09:22.024113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.024088    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:22.090079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.090045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\" (UniqueName: \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\") pod \"e272e37a-f4ca-428e-b9c7-f381eb55b913\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") "
Nov 01 20:09:22.090279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.090125    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\") pod \"e272e37a-f4ca-428e-b9c7-f381eb55b913\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") "
Nov 01 20:09:22.090279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.090162    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\") pod \"e272e37a-f4ca-428e-b9c7-f381eb55b913\" (UID: \"e272e37a-f4ca-428e-b9c7-f381eb55b913\") "
Nov 01 20:09:22.090927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.090887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca" pod "e272e37a-f4ca-428e-b9c7-f381eb55b913" (UID: "e272e37a-f4ca-428e-b9c7-f381eb55b913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:22.092288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.092267    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca" pod "e272e37a-f4ca-428e-b9c7-f381eb55b913" (UID: "e272e37a-f4ca-428e-b9c7-f381eb55b913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:22.092814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.092796    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz" pod "e272e37a-f4ca-428e-b9c7-f381eb55b913" (UID: "e272e37a-f4ca-428e-b9c7-f381eb55b913"). InnerVolumeSpecName "kube-api-access-9jbjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:22.191657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.191609    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:22.191657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.191644    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jbjz\" (UniqueName: \"kubernetes.io/projected/e272e37a-f4ca-428e-b9c7-f381eb55b913-kube-api-access-9jbjz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:22.191657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.191656    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e272e37a-f4ca-428e-b9c7-f381eb55b913-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:22.641723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.641675    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:09:22.641934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.641899    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:22.642018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.641953    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:22.794625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.794593    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"
Nov 01 20:09:22.795111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.794773    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9" event={"ID":"e272e37a-f4ca-428e-b9c7-f381eb55b913","Type":"ContainerDied","Data":"ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c"}
Nov 01 20:09:22.795111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:22.794866    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff27b8e714e90e417aa6c273e41cbef6f54ee8f8f5635cb0a01c919d354f842c"
Nov 01 20:09:23.414930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.414886    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"]
Nov 01 20:09:23.415505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415493    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e272e37a-f4ca-428e-b9c7-f381eb55b913" containerName="revoke-keys"
Nov 01 20:09:23.415554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415507    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e272e37a-f4ca-428e-b9c7-f381eb55b913" containerName="revoke-keys"
Nov 01 20:09:23.415554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415524    5041 cpu_manager.go:414] "containerMap: removing container" podUID="86ec6baa-35a6-4538-bfbd-833185672008" containerName="revoke-keys"
Nov 01 20:09:23.415554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415529    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ec6baa-35a6-4538-bfbd-833185672008" containerName="revoke-keys"
Nov 01 20:09:23.415657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415615    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ec6baa-35a6-4538-bfbd-833185672008" containerName="revoke-keys"
Nov 01 20:09:23.415657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.415631    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e272e37a-f4ca-428e-b9c7-f381eb55b913" containerName="revoke-keys"
Nov 01 20:09:23.419515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.419496    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.423760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.423584    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:23.424828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.424804    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:23.424972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.424850    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:23.424972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.424904    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:23.427185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.427163    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"]
Nov 01 20:09:23.511348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.511304    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.511551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.511437    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6zs\" (UniqueName: \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.511706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.511680    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.612498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.612461    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.612692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.612539    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6zs\" (UniqueName: \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.612692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.612621    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.613232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.613207    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.614856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.614831    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.621225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.621193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6zs\" (UniqueName: \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\") pod \"maas-api-revoke-sub-d280331d-4fthw\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:23.641048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.641026    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:23.641150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.641079    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:23.735333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:23.735256    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:24.043036 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:24.042981    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ee9b06_9cd4_4227_bbfa_ec722ba76d3c.slice/crio-3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684 WatchSource:0}: Error finding container 3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684: Status 404 returned error can't find the container with id 3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684
Nov 01 20:09:24.068036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.067257    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"]
Nov 01 20:09:24.641913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.641758    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:24.641913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.641864    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:24.821515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.821471    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw" event={"ID":"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c","Type":"ContainerStarted","Data":"ce2ec72747f978c03689459d2d67f7fc097bfe8c50330352e61df30614744320"}
Nov 01 20:09:24.821691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.821534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw" event={"ID":"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c","Type":"ContainerStarted","Data":"3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684"}
Nov 01 20:09:24.841382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:24.841332    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw" podStartSLOduration=1.84131701 podStartE2EDuration="1.84131701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:24.835990504 +0000 UTC m=+652.557069175" watchObservedRunningTime="2026-11-01 20:09:24.84131701 +0000 UTC m=+652.562395722"
Nov 01 20:09:25.790804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:25.786657    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:25.790804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:25.786744    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:25.954817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:25.945098    5041 generic.go:356] "Generic (PLEG): container finished" podID="81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" containerID="ce2ec72747f978c03689459d2d67f7fc097bfe8c50330352e61df30614744320" exitCode=0
Nov 01 20:09:25.954817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:25.945166    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw" event={"ID":"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c","Type":"ContainerDied","Data":"ce2ec72747f978c03689459d2d67f7fc097bfe8c50330352e61df30614744320"}
Nov 01 20:09:26.082924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.075380    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:09:26.209458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.180857    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311631    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311684    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhrw\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311801    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311887    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.311933    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.312075    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.312183    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.326815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.312214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.428657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.428564    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.428936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.428911    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.429129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.429111    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.429243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.429229    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.437031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.436994    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.437309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.437289    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.437510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.437492    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.437643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.437628    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.437759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.437739    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhrw\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.466083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.466039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.466629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.466605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.484828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.482458    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.532805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.508279    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.532805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.508617    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.557324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.557288    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.567569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.567502    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.646802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.643963    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:26.646802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.644024    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:26.727860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.717366    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.727860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.718478    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhrw\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:26.766810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.765846    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:09:26.806801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.806287    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:26.842803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.819590    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:26.945803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.931534    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:09:26.945803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.937904    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:26.945803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:26.938156    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.011502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.011007    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022072    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022195    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022223    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022249    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022359    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022450    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022488    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022515    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkv2z\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.022546    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.030899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.026882    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:09:27.038423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.035224    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:09:27.077066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.064664    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.092270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.092236    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.111949    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124217    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124394    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124447    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124579    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124679    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124723    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124759    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkv2z\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.129732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.124813    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.133666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.131629    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.133666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.132326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.133666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.132470    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.133666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.132821    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.135890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.135634    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.151823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.139415    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.189381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.189330    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.192740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.192658    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.196630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.196586    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.197846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.197818    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkv2z\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.200153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.200122    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231019    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231158    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231245    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231284    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231310    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231341    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231372    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.231844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.231571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vg9t\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.329945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.329914    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:27.332821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.332629    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.332821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.332676    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.332821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.332724    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.332868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333015    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vg9t\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333067    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333137    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333162    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333229    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333260    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333286    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333322    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333327    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333378    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333417    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdw9\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333455    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333480    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333497    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333517    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333534    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333606    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.333863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.333752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.334585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.334558    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.335845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.335823    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.336807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.336770    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.374162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.374130    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vg9t\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.374308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.374285    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.377133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.377112    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:27.435530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.435501    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\") pod \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") "
Nov 01 20:09:27.435697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.435682    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\" (UniqueName: \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\") pod \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") "
Nov 01 20:09:27.435769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.435716    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\") pod \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\" (UID: \"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c\") "
Nov 01 20:09:27.436033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436053    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436082    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436154    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436192    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdw9\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436323    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436404    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.436544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.437189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.436933    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.438246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.438138    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.438246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.438182    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca" pod "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" (UID: "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:27.438480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.438460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.439025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.439001    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.439741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.439714    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.440604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.440564    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca" pod "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" (UID: "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:27.441035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.441010    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.441235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.441214    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs" pod "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" (UID: "81ee9b06-9cd4-4227-bbfa-ec722ba76d3c"). InnerVolumeSpecName "kube-api-access-xm6zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:27.460248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.459688    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:27.465132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.465106    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.466645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.466619    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdw9\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.469552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.469525    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:09:27.477913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.477887    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.483296 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:27.483269    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb926a5d_b67e_4eb1_b634_aa07bfee99a5.slice/crio-382cd7f327ac5a4c297835bb45b08c2df5684e155e404d5a1146fc198932b9a3 WatchSource:0}: Error finding container 382cd7f327ac5a4c297835bb45b08c2df5684e155e404d5a1146fc198932b9a3: Status 404 returned error can't find the container with id 382cd7f327ac5a4c297835bb45b08c2df5684e155e404d5a1146fc198932b9a3
Nov 01 20:09:27.485564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.485519    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.485644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.485622    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.485681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.485671    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.509743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.509707    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:27.538673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.538566    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:27.538673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.538606    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6zs\" (UniqueName: \"kubernetes.io/projected/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-kube-api-access-xm6zs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:27.538673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.538623    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:27.644525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.644075    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Nov 01 20:09:27.644525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.644135    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Nov 01 20:09:27.679200 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:27.675110    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b4d85e_e38b_4544_a97a_52dd68367199.slice/crio-29f063176574f07aa8dd7d61c61bd5a8fa3651ff0ee5d9abdbf29c0b6ef2bbba WatchSource:0}: Error finding container 29f063176574f07aa8dd7d61c61bd5a8fa3651ff0ee5d9abdbf29c0b6ef2bbba: Status 404 returned error can't find the container with id 29f063176574f07aa8dd7d61c61bd5a8fa3651ff0ee5d9abdbf29c0b6ef2bbba
Nov 01 20:09:27.679200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.678292    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.679200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.678422    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.679200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.678499    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.690537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.690508    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:09:27.783551 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:27.783494    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec88d3f_2ec4_46f7_9740_8bad25ce1ba2.slice/crio-5d1e68c43febcc98d6e71c65a136fb5f9924816a1c262cd7a858b94860921c50 WatchSource:0}: Error finding container 5d1e68c43febcc98d6e71c65a136fb5f9924816a1c262cd7a858b94860921c50: Status 404 returned error can't find the container with id 5d1e68c43febcc98d6e71c65a136fb5f9924816a1c262cd7a858b94860921c50
Nov 01 20:09:27.787743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.787011    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.787743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.787106    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.787743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.787147    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.789162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.789023    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:09:27.861373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.859319    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:09:27.862568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.862530    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.862845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.862821    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:27.863033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:27.863003    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:28.080397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.079430    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" event={"ID":"fb926a5d-b67e-4eb1-b634-aa07bfee99a5","Type":"ContainerStarted","Data":"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"}
Nov 01 20:09:28.080397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.079486    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" event={"ID":"fb926a5d-b67e-4eb1-b634-aa07bfee99a5","Type":"ContainerStarted","Data":"382cd7f327ac5a4c297835bb45b08c2df5684e155e404d5a1146fc198932b9a3"}
Nov 01 20:09:28.084009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.082590    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" event={"ID":"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2","Type":"ContainerStarted","Data":"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"}
Nov 01 20:09:28.084009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.082633    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" event={"ID":"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2","Type":"ContainerStarted","Data":"5d1e68c43febcc98d6e71c65a136fb5f9924816a1c262cd7a858b94860921c50"}
Nov 01 20:09:28.088026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.087938    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"
Nov 01 20:09:28.089162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.088857    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw" event={"ID":"81ee9b06-9cd4-4227-bbfa-ec722ba76d3c","Type":"ContainerDied","Data":"3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684"}
Nov 01 20:09:28.089162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.088913    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ae0504563047a1628209bdd978a3fcf7f1dafddca6fa6c82a8a61d0eedb3684"
Nov 01 20:09:28.090764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.090729    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" event={"ID":"54b6a89d-9b18-45fd-a952-3fe23382defa","Type":"ContainerStarted","Data":"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"}
Nov 01 20:09:28.097246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.092873    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" event={"ID":"54b6a89d-9b18-45fd-a952-3fe23382defa","Type":"ContainerStarted","Data":"807e7943dd0a59b833a3c3a785dd19ef52a7b4089f0220b9065b7c6df84b3902"}
Nov 01 20:09:28.098072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.098048    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" event={"ID":"68b4d85e-e38b-4544-a97a-52dd68367199","Type":"ContainerStarted","Data":"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"}
Nov 01 20:09:28.098279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.098249    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" event={"ID":"68b4d85e-e38b-4544-a97a-52dd68367199","Type":"ContainerStarted","Data":"29f063176574f07aa8dd7d61c61bd5a8fa3651ff0ee5d9abdbf29c0b6ef2bbba"}
Nov 01 20:09:28.129593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.129534    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podStartSLOduration=2.129514963 podStartE2EDuration="2.129514963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.124632516 +0000 UTC m=+655.845711204" watchObservedRunningTime="2026-11-01 20:09:28.129514963 +0000 UTC m=+655.850593640"
Nov 01 20:09:28.130498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.130451    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podStartSLOduration=2.130437095 podStartE2EDuration="2.130437095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.10381347 +0000 UTC m=+655.824892147" watchObservedRunningTime="2026-11-01 20:09:28.130437095 +0000 UTC m=+655.851515772"
Nov 01 20:09:28.157223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.157153    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podStartSLOduration=2.157134447 podStartE2EDuration="2.157134447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.152318008 +0000 UTC m=+655.873396757" watchObservedRunningTime="2026-11-01 20:09:28.157134447 +0000 UTC m=+655.878213125"
Nov 01 20:09:28.186455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.186397    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podStartSLOduration=2.186380707 podStartE2EDuration="2.186380707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.182038648 +0000 UTC m=+655.903117326" watchObservedRunningTime="2026-11-01 20:09:28.186380707 +0000 UTC m=+655.907459382"
Nov 01 20:09:28.377942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.377859    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:28.378746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.378720    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:28.378844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.378800    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:28.465799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.465339    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:28.465799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.465404    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:28.469801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.467287    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:28.512929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.512855    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:28.653859 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:09:28.651017    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:09:28.684227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.684187    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:09:28.686330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.686300    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:09:28.699336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:28.698804    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:09:29.015858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.015098    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:29.040111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.034197    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.040111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.034277    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:29.453380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.453332    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.453659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.453631    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:29.467855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.467808    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.468139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.468110    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:29.514486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.514450    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:29.514815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:29.514771    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:30.016804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.014246    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.016804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.014326    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:30.391802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.391210    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.391802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.391276    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:30.484832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.484143    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.484832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.484202    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:30.515804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.511146    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:30.515804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.511209    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:30.737530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.718886    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:30.737530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:30.734974    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:31.128805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.114801    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.128805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.114869    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:31.444857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.407414    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.446550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.446514    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:31.471224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.466178    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.495804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.474878    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:31.795802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.792244    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:09:31.795802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.793264    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:09:31.797298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.797267    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:31.798959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.798938    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:09:31.820805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.805260    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:31.820805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.805309    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:31.921804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:31.905868    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:32.056485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.056386    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.056765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.056718    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:32.539392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.539354    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.539981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.539947    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:32.613810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.577243    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.613810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.577300    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:32.760803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.677074    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:32.760803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.677148    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:32.800126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.800028    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:09:32.801453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:32.801181    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:33.039806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.037037    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.039806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.037104    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:33.399813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.397238    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.399813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.397312    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:33.461317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.461273    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.461575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.461543    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:33.518803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.510455    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:33.518803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.510542    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:33.728859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.713292    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:09:33.807943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.807888    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:33.819947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.819891    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Nov 01 20:09:33.847668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.847622    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Nov 01 20:09:33.960800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:33.953308    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:09:34.031860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.030967    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:34.031860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.031031    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:34.384731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.384206    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:34.384731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.384312    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:34.491802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.483814    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:34.491802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.483878    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:34.511207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.511169    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:34.511488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.511454    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:34.806206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.806152    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:34.809888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.809854    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:09:34.955618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.953452    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:34.955618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.954193    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:09:34.986413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:34.986381    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:09:35.048209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.048173    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:35.052911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.052873    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:35.451908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.451873    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:35.456638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.456598    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:35.460989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.460883    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 01 20:09:35.460989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.460936    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 01 20:09:35.498640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.498366    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:09:35.510572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.510540    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 01 20:09:35.510788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.510593    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 01 20:09:35.960021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.959980    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"]
Nov 01 20:09:35.965610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.965577    5041 cpu_manager.go:414] "containerMap: removing container" podUID="81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" containerName="revoke-keys"
Nov 01 20:09:35.965855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.965835    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" containerName="revoke-keys"
Nov 01 20:09:35.966297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.966268    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" containerName="revoke-keys"
Nov 01 20:09:35.972790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:35.972750    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.011805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.011752    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:36.012231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.012209    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:36.012417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.012401    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:36.012627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.012613    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:36.021173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.014727    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.023533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.021664    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 01 20:09:36.023533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.023140    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 01 20:09:36.056230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.056186    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.056863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.056835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8ml\" (UniqueName: \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.071062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.071034    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"]
Nov 01 20:09:36.159248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.158857    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.159248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.158935    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.159248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.159122    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8ml\" (UniqueName: \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.160310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.160281    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.162607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.162559    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.168158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.168129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8ml\" (UniqueName: \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\") pod \"maas-api-revoke-sub-b762708d-rqddq\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.375986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.375884    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:36.378901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.378378    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 01 20:09:36.378901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.378432    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 01 20:09:36.485460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.483760    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:36.489666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.489083    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:36.491112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.491083    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:09:36.691163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.691077    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:36.697841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.692509    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:36.725461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.725431    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:09:36.797818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:36.797485    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"]
Nov 01 20:09:36.863637 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:36.862048    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc800e33c_4ef5_4f98_abaa_48ff15bd6ee0.slice/crio-a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56 WatchSource:0}: Error finding container a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56: Status 404 returned error can't find the container with id a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56
Nov 01 20:09:37.122248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.122211    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:37.123509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.123488    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:37.124439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.124417    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:09:37.226291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.226255    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq" event={"ID":"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0","Type":"ContainerStarted","Data":"a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56"}
Nov 01 20:09:37.383000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.382913    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:37.383462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.383436    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:37.384229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:37.384206    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:09:38.244871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:38.244834    5041 generic.go:356] "Generic (PLEG): container finished" podID="c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" containerID="68ec18dddf24ecce39afc5cc0738953c9254e0b6911988754a2232ab56204e0a" exitCode=0
Nov 01 20:09:38.246682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:38.245510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq" event={"ID":"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0","Type":"ContainerDied","Data":"68ec18dddf24ecce39afc5cc0738953c9254e0b6911988754a2232ab56204e0a"}
Nov 01 20:09:39.812208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:39.812162    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:09:40.011805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:39.995617    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:09:40.536585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.536553    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:40.665854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.648060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\") pod \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") "
Nov 01 20:09:40.665854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.648145    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\") pod \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") "
Nov 01 20:09:40.665854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.648202    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\" (UniqueName: \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\") pod \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\" (UID: \"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0\") "
Nov 01 20:09:40.831294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.831257    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca" pod "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" (UID: "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:40.899534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.890874    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:40.970286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.970200    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca" pod "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" (UID: "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:40.991748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:40.991711    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml" pod "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" (UID: "c800e33c-4ef5-4f98-abaa-48ff15bd6ee0"). InnerVolumeSpecName "kube-api-access-tg8ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:41.005118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:41.003993    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:41.005118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:41.004035    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8ml\" (UniqueName: \"kubernetes.io/projected/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0-kube-api-access-tg8ml\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:41.527145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:41.527095    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq" event={"ID":"c800e33c-4ef5-4f98-abaa-48ff15bd6ee0","Type":"ContainerDied","Data":"a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56"}
Nov 01 20:09:41.527339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:41.527168    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91c5fa91d2c272a7ff96d6761858286981d4c4172722cbacc293a5a38c6bc56"
Nov 01 20:09:41.527339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:41.527120    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"
Nov 01 20:09:46.176824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.176714    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"]
Nov 01 20:09:46.177593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.177574    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" containerName="revoke-keys"
Nov 01 20:09:46.177663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.177595    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" containerName="revoke-keys"
Nov 01 20:09:46.177699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.177680    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" containerName="revoke-keys"
Nov 01 20:09:46.181261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.181231    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.184920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.184891    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:46.185600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.185580    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:46.185751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.185733    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:46.185874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.185861    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:46.189920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.189888    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"]
Nov 01 20:09:46.282520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.282307    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.282520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.282360    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.282898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.282651    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqjv\" (UniqueName: \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.383689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.383655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.383689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.383690    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.383914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.383868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqjv\" (UniqueName: \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.384326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.384303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.386260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.386235    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.393863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.393839    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqjv\" (UniqueName: \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\") pod \"maas-api-revoke-sub-e5362b1e-pmqvn\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.502549 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.502455    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:46.797906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.797862    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"]
Nov 01 20:09:46.800524 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:46.800488    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e35dc3_6727_4aaa_bde6_e1a4f2d9d55b.slice/crio-5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182 WatchSource:0}: Error finding container 5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182: Status 404 returned error can't find the container with id 5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182
Nov 01 20:09:46.807326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.803708    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:46.807326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.805086    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"]
Nov 01 20:09:46.810632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.809806    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"]
Nov 01 20:09:46.899088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.899047    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:46.899237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.899128    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:46.899402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:46.899383    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7s2c\" (UniqueName: \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.001071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.001039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7s2c\" (UniqueName: \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.001274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.001100    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.001274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.001136    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.002010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.001986    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.003710 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.003685    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.011301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.011270    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7s2c\" (UniqueName: \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\") pod \"maas-api-revoke-sub-94716fca-54ddk\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.137508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.137472    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:47.521083 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:47.521051    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf482259a_049f_4625_ba3a_70223cc5852e.slice/crio-985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88 WatchSource:0}: Error finding container 985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88: Status 404 returned error can't find the container with id 985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88
Nov 01 20:09:47.521477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.521396    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"]
Nov 01 20:09:47.586254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.586221    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk" event={"ID":"f482259a-049f-4625-ba3a-70223cc5852e","Type":"ContainerStarted","Data":"985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88"}
Nov 01 20:09:47.587922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.587898    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn" event={"ID":"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b","Type":"ContainerStarted","Data":"1a31533bf27773582aab6aaddd882c9e08961bea81ef52d986bd7fbebeb0f37e"}
Nov 01 20:09:47.588061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.587935    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn" event={"ID":"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b","Type":"ContainerStarted","Data":"5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182"}
Nov 01 20:09:47.609998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.609956    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn" podStartSLOduration=1.609942744 podStartE2EDuration="1.609942744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:47.605295346 +0000 UTC m=+675.326374022" watchObservedRunningTime="2026-11-01 20:09:47.609942744 +0000 UTC m=+675.331021467"
Nov 01 20:09:47.676415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.676333    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:09:47.682294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.682272    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.687617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.687587    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 20:09:47.687617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.687604    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 20:09:47.687831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.687695    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbwtw\""
Nov 01 20:09:47.687945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.687920    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 20:09:47.692111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.692087    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:09:47.837863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.837827    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.837884    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.838016    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwbs\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838178 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.838161    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.838190    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.838214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.838340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.838329    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.939610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939579    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.939822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939614    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.939822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.939822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939701    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.940006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939859    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.940006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939906    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.940006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.939939    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwbs\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.941234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.941203    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.944313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.944286    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.944818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.944722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.944932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.944861    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.945892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.945868    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.954911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.954877    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.955546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.955527    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwbs\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2vf7\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:47.997846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:47.997803    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:48.285970 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:48.285935    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b286be8_2d7d_41cd_89b1_ef48aaec78fc.slice/crio-bf37d90e63a508e7a9ff283209565fbaba0124b75c0fea70f0cd34aa6887f355 WatchSource:0}: Error finding container bf37d90e63a508e7a9ff283209565fbaba0124b75c0fea70f0cd34aa6887f355: Status 404 returned error can't find the container with id bf37d90e63a508e7a9ff283209565fbaba0124b75c0fea70f0cd34aa6887f355
Nov 01 20:09:48.288214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.288197    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:09:48.288684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.288661    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:09:48.598640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.598571    5041 generic.go:356] "Generic (PLEG): container finished" podID="45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" containerID="1a31533bf27773582aab6aaddd882c9e08961bea81ef52d986bd7fbebeb0f37e" exitCode=0
Nov 01 20:09:48.599115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.598657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn" event={"ID":"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b","Type":"ContainerDied","Data":"1a31533bf27773582aab6aaddd882c9e08961bea81ef52d986bd7fbebeb0f37e"}
Nov 01 20:09:48.600752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.600723    5041 generic.go:356] "Generic (PLEG): container finished" podID="f482259a-049f-4625-ba3a-70223cc5852e" containerID="b427b06385e2ccd416cb103a05d5a236b0af4fd5c4aeeb6b56918b4338e40e35" exitCode=0
Nov 01 20:09:48.600938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.600810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk" event={"ID":"f482259a-049f-4625-ba3a-70223cc5852e","Type":"ContainerDied","Data":"b427b06385e2ccd416cb103a05d5a236b0af4fd5c4aeeb6b56918b4338e40e35"}
Nov 01 20:09:48.602506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:48.602483    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" event={"ID":"4b286be8-2d7d-41cd-89b1-ef48aaec78fc","Type":"ContainerStarted","Data":"bf37d90e63a508e7a9ff283209565fbaba0124b75c0fea70f0cd34aa6887f355"}
Nov 01 20:09:49.869424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.869351    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:49.875298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.874790    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969101    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\") pod \"f482259a-049f-4625-ba3a-70223cc5852e\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969162    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\" (UniqueName: \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\") pod \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969217    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\" (UniqueName: \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\") pod \"f482259a-049f-4625-ba3a-70223cc5852e\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969297    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\") pod \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969335    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\") pod \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\" (UID: \"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.969380    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\") pod \"f482259a-049f-4625-ba3a-70223cc5852e\" (UID: \"f482259a-049f-4625-ba3a-70223cc5852e\") "
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.970369    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca" pod "f482259a-049f-4625-ba3a-70223cc5852e" (UID: "f482259a-049f-4625-ba3a-70223cc5852e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:49.974837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.973090    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca" pod "f482259a-049f-4625-ba3a-70223cc5852e" (UID: "f482259a-049f-4625-ba3a-70223cc5852e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:49.979995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.979949    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca" pod "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" (UID: "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:09:49.996844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.993074    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv" pod "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" (UID: "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b"). InnerVolumeSpecName "kube-api-access-mfqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:49.996844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.993151    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca" pod "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" (UID: "45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:09:49.996844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:49.994903    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c" pod "f482259a-049f-4625-ba3a-70223cc5852e" (UID: "f482259a-049f-4625-ba3a-70223cc5852e"). InnerVolumeSpecName "kube-api-access-q7s2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:09:50.071317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071279    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.071317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071319    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.071524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071333    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f482259a-049f-4625-ba3a-70223cc5852e-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.071524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071346    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqjv\" (UniqueName: \"kubernetes.io/projected/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-kube-api-access-mfqjv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.071524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071360    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7s2c\" (UniqueName: \"kubernetes.io/projected/f482259a-049f-4625-ba3a-70223cc5852e-kube-api-access-q7s2c\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.071524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.071373    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:09:50.626361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.626321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn" event={"ID":"45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b","Type":"ContainerDied","Data":"5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182"}
Nov 01 20:09:50.626551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.626375    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fcf813d676bc5d42438a56aadb9fc5d85ec9924fb085fe0a04bca188767d182"
Nov 01 20:09:50.626551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.626450    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"
Nov 01 20:09:50.629402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.628820    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"
Nov 01 20:09:50.629402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.628851    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk" event={"ID":"f482259a-049f-4625-ba3a-70223cc5852e","Type":"ContainerDied","Data":"985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88"}
Nov 01 20:09:50.629402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:50.628900    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985306799e8a32e08585865baaac9cc3b82c10800f7a06fd073674041b90bb88"
Nov 01 20:09:52.633584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:52.633549    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:52.633900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:52.633618    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:09:53.675736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:53.675701    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" event={"ID":"4b286be8-2d7d-41cd-89b1-ef48aaec78fc","Type":"ContainerStarted","Data":"e14ca4e23a101115c90470df8ac9d2b411a825802de1e2fd5323b1f5b32930ec"}
Nov 01 20:09:53.676234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:53.676007    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:53.678061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:53.678038    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:09:53.696565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:53.696513    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" podStartSLOduration=2.352494072 podStartE2EDuration="6.696501017s" totalImagesPullingTime="4.344006945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:53.694614525 +0000 UTC m=+681.415693204" watchObservedRunningTime="2026-11-01 20:09:53.696501017 +0000 UTC m=+681.417579692"
Nov 01 20:09:57.748924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.748889    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"]
Nov 01 20:09:57.749552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749536    5041 cpu_manager.go:414] "containerMap: removing container" podUID="45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" containerName="revoke-keys"
Nov 01 20:09:57.749606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749553    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" containerName="revoke-keys"
Nov 01 20:09:57.749606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749563    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f482259a-049f-4625-ba3a-70223cc5852e" containerName="revoke-keys"
Nov 01 20:09:57.749606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749568    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f482259a-049f-4625-ba3a-70223cc5852e" containerName="revoke-keys"
Nov 01 20:09:57.749701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749655    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" containerName="revoke-keys"
Nov 01 20:09:57.749701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.749669    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f482259a-049f-4625-ba3a-70223cc5852e" containerName="revoke-keys"
Nov 01 20:09:57.753621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.753601    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:57.758865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.758840    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:09:57.760530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.760505    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:09:57.761155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.761131    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"]
Nov 01 20:09:57.853465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.853435    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:09:57.853615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.853494    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:09:57.903595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.903555    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:57.903812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.903754    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:57.903895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:57.903849    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7br\" (UniqueName: \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.017843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.011687    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.017843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.011798    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7br\" (UniqueName: \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.017843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.011889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.017843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.013065    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.035565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.035534    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7br\" (UniqueName: \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.046187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.046150    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc32c16c-gbs9j\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.077014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.076979    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:09:58.366407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.366374    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"]
Nov 01 20:09:58.372184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.372158    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.386245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.386215    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"]
Nov 01 20:09:58.465984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.465953    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"]
Nov 01 20:09:58.466555 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:58.466526    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb685d55_1973_4d42_be9e_6d6f0dd9bff8.slice/crio-af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2 WatchSource:0}: Error finding container af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2: Status 404 returned error can't find the container with id af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2
Nov 01 20:09:58.538046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.538010    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cwk\" (UniqueName: \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.538337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.538315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.538555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.538532    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.541753 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:09:58.541720    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:09:58.639586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.639495    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cwk\" (UniqueName: \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.639586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.639577    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.639861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.639631    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.640346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.640319    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.641974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.641952    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.647560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.647538    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cwk\" (UniqueName: \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\") pod \"maas-api-revoke-sub-cb9c6264-bv9s8\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.689982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.689948    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:09:58.717338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.717309    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j" event={"ID":"bb685d55-1973-4d42-be9e-6d6f0dd9bff8","Type":"ContainerStarted","Data":"af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2"}
Nov 01 20:09:58.976241 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:09:58.976211    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f44b95_f366_40ef_9895_69b6ca1bbd7a.slice/crio-6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33 WatchSource:0}: Error finding container 6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33: Status 404 returned error can't find the container with id 6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33
Nov 01 20:09:58.976883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:58.976862    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"]
Nov 01 20:09:59.730198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:59.730156    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8" event={"ID":"35f44b95-f366-40ef-9895-69b6ca1bbd7a","Type":"ContainerStarted","Data":"684b4c106abda7bb0a36c12c1f98a18fce554de842e0c24558889526f94da909"}
Nov 01 20:09:59.730198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:59.730221    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8" event={"ID":"35f44b95-f366-40ef-9895-69b6ca1bbd7a","Type":"ContainerStarted","Data":"6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33"}
Nov 01 20:09:59.732014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:59.731982    5041 generic.go:356] "Generic (PLEG): container finished" podID="bb685d55-1973-4d42-be9e-6d6f0dd9bff8" containerID="10dac9d7d9cf6d5dd4104d7d840a78f15222e0b3f109f7f90be89a9513e182fa" exitCode=0
Nov 01 20:09:59.732146 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:59.732066    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j" event={"ID":"bb685d55-1973-4d42-be9e-6d6f0dd9bff8","Type":"ContainerDied","Data":"10dac9d7d9cf6d5dd4104d7d840a78f15222e0b3f109f7f90be89a9513e182fa"}
Nov 01 20:09:59.748840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:09:59.748794    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8" podStartSLOduration=1.748765008 podStartE2EDuration="1.748765008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:59.744439974 +0000 UTC m=+687.465518653" watchObservedRunningTime="2026-11-01 20:09:59.748765008 +0000 UTC m=+687.469843683"
Nov 01 20:10:00.742359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:00.742318    5041 generic.go:356] "Generic (PLEG): container finished" podID="35f44b95-f366-40ef-9895-69b6ca1bbd7a" containerID="684b4c106abda7bb0a36c12c1f98a18fce554de842e0c24558889526f94da909" exitCode=0
Nov 01 20:10:00.742840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:00.742460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8" event={"ID":"35f44b95-f366-40ef-9895-69b6ca1bbd7a","Type":"ContainerDied","Data":"684b4c106abda7bb0a36c12c1f98a18fce554de842e0c24558889526f94da909"}
Nov 01 20:10:00.905887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:00.905862    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:10:01.074510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.074473    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\") pod \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") "
Nov 01 20:10:01.074693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.074606    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\" (UniqueName: \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\") pod \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") "
Nov 01 20:10:01.074829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.074812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\") pod \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\" (UID: \"bb685d55-1973-4d42-be9e-6d6f0dd9bff8\") "
Nov 01 20:10:01.075092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.075067    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca" pod "bb685d55-1973-4d42-be9e-6d6f0dd9bff8" (UID: "bb685d55-1973-4d42-be9e-6d6f0dd9bff8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:01.075293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.075277    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:01.076663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.076644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br" pod "bb685d55-1973-4d42-be9e-6d6f0dd9bff8" (UID: "bb685d55-1973-4d42-be9e-6d6f0dd9bff8"). InnerVolumeSpecName "kube-api-access-sc7br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:01.076741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.076723    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca" pod "bb685d55-1973-4d42-be9e-6d6f0dd9bff8" (UID: "bb685d55-1973-4d42-be9e-6d6f0dd9bff8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:01.176655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.176619    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:01.176655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.176654    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7br\" (UniqueName: \"kubernetes.io/projected/bb685d55-1973-4d42-be9e-6d6f0dd9bff8-kube-api-access-sc7br\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:01.755559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.755532    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"
Nov 01 20:10:01.755559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.755534    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j" event={"ID":"bb685d55-1973-4d42-be9e-6d6f0dd9bff8","Type":"ContainerDied","Data":"af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2"}
Nov 01 20:10:01.756314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.755591    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af58f3a8e4883e410aa993cd87ea7d093d6a30ce14d8be3f2744a6d1e28f1fd2"
Nov 01 20:10:01.933962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:01.933930    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:10:02.088731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.088689    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\" (UniqueName: \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\") pod \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") "
Nov 01 20:10:02.088962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.088831    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\") pod \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") "
Nov 01 20:10:02.088962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.088866    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\") pod \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\" (UID: \"35f44b95-f366-40ef-9895-69b6ca1bbd7a\") "
Nov 01 20:10:02.089304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.089279    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca" pod "35f44b95-f366-40ef-9895-69b6ca1bbd7a" (UID: "35f44b95-f366-40ef-9895-69b6ca1bbd7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:02.089563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.089541    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:02.090928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.090898    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk" pod "35f44b95-f366-40ef-9895-69b6ca1bbd7a" (UID: "35f44b95-f366-40ef-9895-69b6ca1bbd7a"). InnerVolumeSpecName "kube-api-access-c6cwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:02.091025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.090958    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca" pod "35f44b95-f366-40ef-9895-69b6ca1bbd7a" (UID: "35f44b95-f366-40ef-9895-69b6ca1bbd7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:02.191410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.191364    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6cwk\" (UniqueName: \"kubernetes.io/projected/35f44b95-f366-40ef-9895-69b6ca1bbd7a-kube-api-access-c6cwk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:02.191410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.191406    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35f44b95-f366-40ef-9895-69b6ca1bbd7a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:02.766469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.766379    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"
Nov 01 20:10:02.766469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.766408    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8" event={"ID":"35f44b95-f366-40ef-9895-69b6ca1bbd7a","Type":"ContainerDied","Data":"6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33"}
Nov 01 20:10:02.766469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:02.766455    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da38b7ebf70dd4cf66f293dc5f1f6f45d81fee9f289cd76b608663902c0dd33"
Nov 01 20:10:04.826893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:04.812290    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:10:04.845456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:04.815823    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:05.000667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:05.000620    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:14.122191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122156    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"]
Nov 01 20:10:14.122803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122787    5041 cpu_manager.go:414] "containerMap: removing container" podUID="35f44b95-f366-40ef-9895-69b6ca1bbd7a" containerName="revoke-keys"
Nov 01 20:10:14.122859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122804    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f44b95-f366-40ef-9895-69b6ca1bbd7a" containerName="revoke-keys"
Nov 01 20:10:14.122859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122840    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bb685d55-1973-4d42-be9e-6d6f0dd9bff8" containerName="revoke-keys"
Nov 01 20:10:14.122859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122845    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb685d55-1973-4d42-be9e-6d6f0dd9bff8" containerName="revoke-keys"
Nov 01 20:10:14.122977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122967    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f44b95-f366-40ef-9895-69b6ca1bbd7a" containerName="revoke-keys"
Nov 01 20:10:14.123015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.122979    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb685d55-1973-4d42-be9e-6d6f0dd9bff8" containerName="revoke-keys"
Nov 01 20:10:14.129649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.129625    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.133520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.132989    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:14.134997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.134706    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:14.134997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.134884    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:14.135178 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.135016    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:14.153229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.153180    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"]
Nov 01 20:10:14.180492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.180453    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.180676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.180501    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjvq\" (UniqueName: \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.180676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.180582    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.281853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.281810    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.281853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.281853    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjvq\" (UniqueName: \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.282084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.282061    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.282708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.282684    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.284495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.284472    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.297013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.296983    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjvq\" (UniqueName: \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\") pod \"maas-api-revoke-sub-da89eacc-ntm6l\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.472846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.472709    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:14.818801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:14.814648    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"]
Nov 01 20:10:15.144815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:15.142848    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l" event={"ID":"b31cdef5-7629-4ee2-8167-38ebb54362be","Type":"ContainerStarted","Data":"41e00b4e319b1c5464444abc85f522bf4e709b54aee38095b9557f2842140fce"}
Nov 01 20:10:16.210623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:16.210567    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l" event={"ID":"b31cdef5-7629-4ee2-8167-38ebb54362be","Type":"ContainerStarted","Data":"d077fb66afd4dc06a66cdc528c656437600f5064be624274bb06e6ad3fc247ae"}
Nov 01 20:10:16.590813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:16.587744    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l" podStartSLOduration=2.587719815 podStartE2EDuration="2.587719815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:16.242984684 +0000 UTC m=+703.964063363" watchObservedRunningTime="2026-11-01 20:10:16.587719815 +0000 UTC m=+704.308798492"
Nov 01 20:10:17.270802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:17.270657    5041 generic.go:356] "Generic (PLEG): container finished" podID="b31cdef5-7629-4ee2-8167-38ebb54362be" containerID="d077fb66afd4dc06a66cdc528c656437600f5064be624274bb06e6ad3fc247ae" exitCode=0
Nov 01 20:10:17.271286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:17.270909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l" event={"ID":"b31cdef5-7629-4ee2-8167-38ebb54362be","Type":"ContainerDied","Data":"d077fb66afd4dc06a66cdc528c656437600f5064be624274bb06e6ad3fc247ae"}
Nov 01 20:10:18.457739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.457711    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:18.552175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.552147    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\") pod \"b31cdef5-7629-4ee2-8167-38ebb54362be\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") "
Nov 01 20:10:18.552337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.552195    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\") pod \"b31cdef5-7629-4ee2-8167-38ebb54362be\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") "
Nov 01 20:10:18.552337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.552312    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\" (UniqueName: \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\") pod \"b31cdef5-7629-4ee2-8167-38ebb54362be\" (UID: \"b31cdef5-7629-4ee2-8167-38ebb54362be\") "
Nov 01 20:10:18.552604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.552572    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca" pod "b31cdef5-7629-4ee2-8167-38ebb54362be" (UID: "b31cdef5-7629-4ee2-8167-38ebb54362be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:18.552815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.552764    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:18.554218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.554191    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca" pod "b31cdef5-7629-4ee2-8167-38ebb54362be" (UID: "b31cdef5-7629-4ee2-8167-38ebb54362be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:18.554415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.554393    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq" pod "b31cdef5-7629-4ee2-8167-38ebb54362be" (UID: "b31cdef5-7629-4ee2-8167-38ebb54362be"). InnerVolumeSpecName "kube-api-access-4bjvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:18.654127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.654090    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31cdef5-7629-4ee2-8167-38ebb54362be-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:18.654127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:18.654121    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjvq\" (UniqueName: \"kubernetes.io/projected/b31cdef5-7629-4ee2-8167-38ebb54362be-kube-api-access-4bjvq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:19.300901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:19.300861    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l" event={"ID":"b31cdef5-7629-4ee2-8167-38ebb54362be","Type":"ContainerDied","Data":"41e00b4e319b1c5464444abc85f522bf4e709b54aee38095b9557f2842140fce"}
Nov 01 20:10:19.301172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:19.301138    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e00b4e319b1c5464444abc85f522bf4e709b54aee38095b9557f2842140fce"
Nov 01 20:10:19.301323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:19.301229    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"
Nov 01 20:10:21.669247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.669189    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"]
Nov 01 20:10:21.670005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.669989    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b31cdef5-7629-4ee2-8167-38ebb54362be" containerName="revoke-keys"
Nov 01 20:10:21.670079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.670006    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31cdef5-7629-4ee2-8167-38ebb54362be" containerName="revoke-keys"
Nov 01 20:10:21.670152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.670143    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31cdef5-7629-4ee2-8167-38ebb54362be" containerName="revoke-keys"
Nov 01 20:10:21.677982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.677956    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.680311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.680281    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"]
Nov 01 20:10:21.681486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.681389    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:21.682490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.682454    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:21.682591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.682519    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:21.682719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.682702    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:21.808876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.808836    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.809070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.809049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz89z\" (UniqueName: \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.809146 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.809131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.911009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.910975    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz89z\" (UniqueName: \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.911185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.911043    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.911236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.911215    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.911874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.911849    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.913432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.913411    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.921941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.921883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz89z\" (UniqueName: \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\") pod \"maas-api-revoke-sub-601ecc0a-f9rz6\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:21.999053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:21.999026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:22.318081 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:10:22.318047    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3981c54_c9bf_4d74_a9c1_f77c98c81a07.slice/crio-fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07 WatchSource:0}: Error finding container fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07: Status 404 returned error can't find the container with id fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07
Nov 01 20:10:22.318445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:22.318422    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"]
Nov 01 20:10:22.346397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:22.346362    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6" event={"ID":"a3981c54-c9bf-4d74-a9c1-f77c98c81a07","Type":"ContainerStarted","Data":"fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07"}
Nov 01 20:10:22.901299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:22.901269    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:10:22.901930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:22.901893    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy" containerID="cri-o://5ecbee747b49e4f30e2af5d47f7c75a81bc0e8f10198cb49eafc5a216c3a263c" gracePeriod=30
Nov 01 20:10:23.357744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:23.357707    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6" event={"ID":"a3981c54-c9bf-4d74-a9c1-f77c98c81a07","Type":"ContainerStarted","Data":"ca9cee8ef80f92087d867a39d2678b0e37032d68860280521cd3e6bc87cc5346"}
Nov 01 20:10:23.375181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:23.375135    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6" podStartSLOduration=2.375123143 podStartE2EDuration="2.375123143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:23.373306849 +0000 UTC m=+711.094385548" watchObservedRunningTime="2026-11-01 20:10:23.375123143 +0000 UTC m=+711.096201818"
Nov 01 20:10:24.367944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:24.367847    5041 generic.go:356] "Generic (PLEG): container finished" podID="a3981c54-c9bf-4d74-a9c1-f77c98c81a07" containerID="ca9cee8ef80f92087d867a39d2678b0e37032d68860280521cd3e6bc87cc5346" exitCode=0
Nov 01 20:10:24.367944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:24.367898    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6" event={"ID":"a3981c54-c9bf-4d74-a9c1-f77c98c81a07","Type":"ContainerDied","Data":"ca9cee8ef80f92087d867a39d2678b0e37032d68860280521cd3e6bc87cc5346"}
Nov 01 20:10:25.546598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.546568    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:25.671492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.671410    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\" (UniqueName: \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\") pod \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") "
Nov 01 20:10:25.671492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.671482    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\") pod \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") "
Nov 01 20:10:25.671679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.671643    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\") pod \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\" (UID: \"a3981c54-c9bf-4d74-a9c1-f77c98c81a07\") "
Nov 01 20:10:25.671992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.671967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca" pod "a3981c54-c9bf-4d74-a9c1-f77c98c81a07" (UID: "a3981c54-c9bf-4d74-a9c1-f77c98c81a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:25.672223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.672204    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:25.673543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.673519    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca" pod "a3981c54-c9bf-4d74-a9c1-f77c98c81a07" (UID: "a3981c54-c9bf-4d74-a9c1-f77c98c81a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:25.673543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.673532    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z" pod "a3981c54-c9bf-4d74-a9c1-f77c98c81a07" (UID: "a3981c54-c9bf-4d74-a9c1-f77c98c81a07"). InnerVolumeSpecName "kube-api-access-lz89z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:25.773671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.773631    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz89z\" (UniqueName: \"kubernetes.io/projected/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-kube-api-access-lz89z\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:25.773671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:25.773664    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3981c54-c9bf-4d74-a9c1-f77c98c81a07-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:26.173231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.173192    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"]
Nov 01 20:10:26.173987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.173967    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a3981c54-c9bf-4d74-a9c1-f77c98c81a07" containerName="revoke-keys"
Nov 01 20:10:26.173987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.173987    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3981c54-c9bf-4d74-a9c1-f77c98c81a07" containerName="revoke-keys"
Nov 01 20:10:26.174113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.174096    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3981c54-c9bf-4d74-a9c1-f77c98c81a07" containerName="revoke-keys"
Nov 01 20:10:26.178995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.178972    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.185268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.185240    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"]
Nov 01 20:10:26.284271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.283445    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ncs\" (UniqueName: \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.284271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.283761    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.284271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.283830    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.386055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.386015    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.386319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.386071    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.386319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.386168    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ncs\" (UniqueName: \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.387816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.387363    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.390084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.390052    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.404419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.404370    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6" event={"ID":"a3981c54-c9bf-4d74-a9c1-f77c98c81a07","Type":"ContainerDied","Data":"fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07"}
Nov 01 20:10:26.404647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.404438    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbae87e43d4a56f06086afd58d6c5a6753ab8342cd3e66a30f19db1e5732d07"
Nov 01 20:10:26.404647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.404507    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"
Nov 01 20:10:26.445163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.445133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ncs\" (UniqueName: \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\") pod \"maas-api-revoke-sub-50c82734-kwdvq\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.499961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.499923    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:26.796841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:26.796754    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"]
Nov 01 20:10:27.415621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:27.415522    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq" event={"ID":"3b819a3c-1b13-486a-8784-e7472f9ab3b3","Type":"ContainerStarted","Data":"d28531e26af7bb1b3cc09472a678ad70f27c9e340f34662c8065219c58379e32"}
Nov 01 20:10:27.415621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:27.415584    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq" event={"ID":"3b819a3c-1b13-486a-8784-e7472f9ab3b3","Type":"ContainerStarted","Data":"38572d54ba3d94a4f2bacf90c96dda506712abecfd3bef68b7f231e8db6664c9"}
Nov 01 20:10:27.445238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:27.445174    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq" podStartSLOduration=1.445152682 podStartE2EDuration="1.445152682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:27.437947605 +0000 UTC m=+715.159026281" watchObservedRunningTime="2026-11-01 20:10:27.445152682 +0000 UTC m=+715.166231360"
Nov 01 20:10:28.730443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:28.730401    5041 generic.go:356] "Generic (PLEG): container finished" podID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerID="5ecbee747b49e4f30e2af5d47f7c75a81bc0e8f10198cb49eafc5a216c3a263c" exitCode=0
Nov 01 20:10:29.421844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:28.733460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerDied","Data":"5ecbee747b49e4f30e2af5d47f7c75a81bc0e8f10198cb49eafc5a216c3a263c"}
Nov 01 20:10:29.422743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.422710    5041 scope.go:122] "RemoveContainer" containerID="feae474dd2e5f66d690605410807ae6f4587cd72931aa13abaf1d4be70ab1060"
Nov 01 20:10:29.426053 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:10:29.426016    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:10:29.476150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.476114    5041 generic.go:356] "Generic (PLEG): container finished" podID="3b819a3c-1b13-486a-8784-e7472f9ab3b3" containerID="d28531e26af7bb1b3cc09472a678ad70f27c9e340f34662c8065219c58379e32" exitCode=0
Nov 01 20:10:29.476577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.476548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq" event={"ID":"3b819a3c-1b13-486a-8784-e7472f9ab3b3","Type":"ContainerDied","Data":"d28531e26af7bb1b3cc09472a678ad70f27c9e340f34662c8065219c58379e32"}
Nov 01 20:10:29.743876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.743836    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:10:29.812806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.811872    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:29.812806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.812214    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:29.847542 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.814252    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:29.898937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.898902    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.002306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.002215    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.142931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.142892    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.155943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.155908    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.207937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.207896    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.209618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.209591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.239052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.239007    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.267862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.240195    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\") pod \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\" (UID: \"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b\") "
Nov 01 20:10:30.268828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:29.999142    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:30.287802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.008952    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:30.288511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.023967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:30.309878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.064160    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:30.310130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.144297    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:30.310251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.208349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:30.310345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.240135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:30.310439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.252528    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:30.337258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.337128    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:10:30.337863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.337750    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g" pod "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" (UID: "b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b"). InnerVolumeSpecName "kube-api-access-rl68g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.374985    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375024    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375043    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375057    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375073    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375087    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375101    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375118    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.386819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.375133    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl68g\" (UniqueName: \"kubernetes.io/projected/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b-kube-api-access-rl68g\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:30.700181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.700148    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"
Nov 01 20:10:30.709258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.709222    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq" event={"ID":"b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b","Type":"ContainerDied","Data":"3e2ef6044704c035a3e3269c0860a8946c765c0854ea1ed9edfd9b838e01fa11"}
Nov 01 20:10:30.709521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.709502    5041 scope.go:122] "RemoveContainer" containerID="5ecbee747b49e4f30e2af5d47f7c75a81bc0e8f10198cb49eafc5a216c3a263c"
Nov 01 20:10:30.791266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.791205    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:10:30.827316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.812864    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:30.827316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.813248    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:10:30.827316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.813526    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:10:30.827316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.813578    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:10:30.865084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:30.864714    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e3c4a7-openshift-default-77d45d48b9-ns4lq"]
Nov 01 20:10:31.333803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:31.332869    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:31.350307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:31.350272    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:10:31.818977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:31.816597    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:10:31.818977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:31.817016    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:32.325609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.278393    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:10:32.571803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.557638    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:32.571803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.561274    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" path="/var/lib/kubelet/pods/b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b/volumes"
Nov 01 20:10:32.571803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.561774    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:10:32.571803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.561837    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:10:32.756395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.756353    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\") pod \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") "
Nov 01 20:10:32.756564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.756548    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\") pod \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") "
Nov 01 20:10:32.756626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.756602    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\" (UniqueName: \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\") pod \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\" (UID: \"3b819a3c-1b13-486a-8784-e7472f9ab3b3\") "
Nov 01 20:10:32.757063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.757033    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca" pod "3b819a3c-1b13-486a-8784-e7472f9ab3b3" (UID: "3b819a3c-1b13-486a-8784-e7472f9ab3b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:32.757282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.757234    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:32.759023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.758997    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca" pod "3b819a3c-1b13-486a-8784-e7472f9ab3b3" (UID: "3b819a3c-1b13-486a-8784-e7472f9ab3b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:32.759277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.759253    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs" pod "3b819a3c-1b13-486a-8784-e7472f9ab3b3" (UID: "3b819a3c-1b13-486a-8784-e7472f9ab3b3"). InnerVolumeSpecName "kube-api-access-69ncs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:32.852241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.852160    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"
Nov 01 20:10:32.852754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.852161    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq" event={"ID":"3b819a3c-1b13-486a-8784-e7472f9ab3b3","Type":"ContainerDied","Data":"38572d54ba3d94a4f2bacf90c96dda506712abecfd3bef68b7f231e8db6664c9"}
Nov 01 20:10:32.852754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.852283    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38572d54ba3d94a4f2bacf90c96dda506712abecfd3bef68b7f231e8db6664c9"
Nov 01 20:10:32.858105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.858079    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b819a3c-1b13-486a-8784-e7472f9ab3b3-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:32.858105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:32.858104    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69ncs\" (UniqueName: \"kubernetes.io/projected/3b819a3c-1b13-486a-8784-e7472f9ab3b3-kube-api-access-69ncs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:34.314685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.314651    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"]
Nov 01 20:10:34.315463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315442    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:34.315551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315465    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:34.315551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315484    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3b819a3c-1b13-486a-8784-e7472f9ab3b3" containerName="revoke-keys"
Nov 01 20:10:34.315551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315489    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b819a3c-1b13-486a-8784-e7472f9ab3b3" containerName="revoke-keys"
Nov 01 20:10:34.315551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315501    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:34.315551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315506    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:34.315709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315655    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:34.315709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.315669    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b819a3c-1b13-486a-8784-e7472f9ab3b3" containerName="revoke-keys"
Nov 01 20:10:34.321322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.321296    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.324979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.324954    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"]
Nov 01 20:10:34.325966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.325942    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:34.325966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.325948    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:34.326843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.326821    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:34.326955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.326879    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:34.388748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.388710    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.388976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.388937    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.389097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.389011    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v9q\" (UniqueName: \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.490188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.490150    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.490509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.490483    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.490683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.490663    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v9q\" (UniqueName: \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.490963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.490942    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.495543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.495511    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.499366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.499340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v9q\" (UniqueName: \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\") pod \"maas-api-revoke-sub-33f2b979-dt6rz\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.639989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.639908    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:34.921723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:34.921696    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"]
Nov 01 20:10:34.922153 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:10:34.922124    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d44a85_a3bf_4820_8208_d9e50665b218.slice/crio-e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d WatchSource:0}: Error finding container e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d: Status 404 returned error can't find the container with id e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d
Nov 01 20:10:35.894948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:35.894906    5041 generic.go:356] "Generic (PLEG): container finished" podID="80d44a85-a3bf-4820-8208-d9e50665b218" containerID="b554b04925dfb12a38512cde9eaf2897e360f114d2bff114b784c034ef25ef6f" exitCode=0
Nov 01 20:10:35.895425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:35.895040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz" event={"ID":"80d44a85-a3bf-4820-8208-d9e50665b218","Type":"ContainerDied","Data":"b554b04925dfb12a38512cde9eaf2897e360f114d2bff114b784c034ef25ef6f"}
Nov 01 20:10:35.895425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:35.895092    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz" event={"ID":"80d44a85-a3bf-4820-8208-d9e50665b218","Type":"ContainerStarted","Data":"e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d"}
Nov 01 20:10:36.578668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:36.578630    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:10:36.578960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:36.578931    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy" containerID="cri-o://e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064" gracePeriod=30
Nov 01 20:10:37.083857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.083829    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:37.148342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.148310    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\" (UniqueName: \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\") pod \"80d44a85-a3bf-4820-8208-d9e50665b218\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") "
Nov 01 20:10:37.148520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.148380    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\") pod \"80d44a85-a3bf-4820-8208-d9e50665b218\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") "
Nov 01 20:10:37.148520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.148459    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\") pod \"80d44a85-a3bf-4820-8208-d9e50665b218\" (UID: \"80d44a85-a3bf-4820-8208-d9e50665b218\") "
Nov 01 20:10:37.149207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.149180    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca" pod "80d44a85-a3bf-4820-8208-d9e50665b218" (UID: "80d44a85-a3bf-4820-8208-d9e50665b218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:37.151253 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.151225    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca" pod "80d44a85-a3bf-4820-8208-d9e50665b218" (UID: "80d44a85-a3bf-4820-8208-d9e50665b218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:37.151253 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.151236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q" pod "80d44a85-a3bf-4820-8208-d9e50665b218" (UID: "80d44a85-a3bf-4820-8208-d9e50665b218"). InnerVolumeSpecName "kube-api-access-z8v9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:37.249958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.249910    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:37.249958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.249950    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v9q\" (UniqueName: \"kubernetes.io/projected/80d44a85-a3bf-4820-8208-d9e50665b218-kube-api-access-z8v9q\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:37.250149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.249966    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d44a85-a3bf-4820-8208-d9e50665b218-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:37.915817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.915738    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz" event={"ID":"80d44a85-a3bf-4820-8208-d9e50665b218","Type":"ContainerDied","Data":"e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d"}
Nov 01 20:10:37.915993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.915846    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d61989df333b37d8a53c2c81d0aaaf156995b153e712ffa28c83e13b1b8f8d"
Nov 01 20:10:37.915993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:37.915921    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"
Nov 01 20:10:38.985172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.980797    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"]
Nov 01 20:10:38.985172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.982252    5041 cpu_manager.go:414] "containerMap: removing container" podUID="80d44a85-a3bf-4820-8208-d9e50665b218" containerName="revoke-keys"
Nov 01 20:10:38.985172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.982273    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d44a85-a3bf-4820-8208-d9e50665b218" containerName="revoke-keys"
Nov 01 20:10:38.985172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.983253    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6aafe53-3043-4cec-a3b4-8b1dfc5b5a1b" containerName="istio-proxy"
Nov 01 20:10:38.985172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.983277    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d44a85-a3bf-4820-8208-d9e50665b218" containerName="revoke-keys"
Nov 01 20:10:38.989224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.989195    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:38.992437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.992406    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"]
Nov 01 20:10:38.998496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.998470    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:38.998636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.998559    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:38.999330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.999272    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:38.999613 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:38.999591    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:39.123699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.123659    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.123917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.123880    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.124014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.123999    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84qg\" (UniqueName: \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.224745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.224715    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.224918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.224837    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.224918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.224891    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84qg\" (UniqueName: \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.225530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.225507    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.227201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.227176    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.232552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.232527    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84qg\" (UniqueName: \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\") pod \"maas-api-revoke-sub-b80ba8aa-tvjgs\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.314941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.314902    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:39.595708 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:10:39.595152    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a7cbca_51b4_48a5_9d4e_1991b98068ad.slice/crio-cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8 WatchSource:0}: Error finding container cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8: Status 404 returned error can't find the container with id cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8
Nov 01 20:10:39.597500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.597472    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"]
Nov 01 20:10:39.956933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:39.956893    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs" event={"ID":"a1a7cbca-51b4-48a5-9d4e-1991b98068ad","Type":"ContainerStarted","Data":"cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8"}
Nov 01 20:10:40.966531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:40.966443    5041 generic.go:356] "Generic (PLEG): container finished" podID="a1a7cbca-51b4-48a5-9d4e-1991b98068ad" containerID="0dc1c84b82a163a3291a2a317c1ca3b70a3bf44b739ba9758875f92e62c35344" exitCode=0
Nov 01 20:10:40.966531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:40.966482    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs" event={"ID":"a1a7cbca-51b4-48a5-9d4e-1991b98068ad","Type":"ContainerDied","Data":"0dc1c84b82a163a3291a2a317c1ca3b70a3bf44b739ba9758875f92e62c35344"}
Nov 01 20:10:41.900883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.900383    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:10:41.986858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.986390    5041 generic.go:356] "Generic (PLEG): container finished" podID="15527bc2-40d5-4af4-9eb0-332391978340" containerID="e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064" exitCode=0
Nov 01 20:10:41.986858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.986459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerDied","Data":"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"}
Nov 01 20:10:41.986858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.986524    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx" event={"ID":"15527bc2-40d5-4af4-9eb0-332391978340","Type":"ContainerDied","Data":"5d9eef236a42b926fd108869f1591e779e75800f3e02e57f972ab2b6096b07cd"}
Nov 01 20:10:41.986858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.986553    5041 scope.go:122] "RemoveContainer" containerID="e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"
Nov 01 20:10:41.987508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:41.987040    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"
Nov 01 20:10:42.001621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.001586    5041 scope.go:122] "RemoveContainer" containerID="d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"
Nov 01 20:10:42.015740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.015714    5041 scope.go:122] "RemoveContainer" containerID="e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"
Nov 01 20:10:42.016126 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:10:42.016103    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064\": container with ID starting with e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064 not found: ID does not exist" containerID="e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"
Nov 01 20:10:42.016238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.016146    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064"} err="failed to get container status \"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064\": rpc error: code = NotFound desc = could not find container \"e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064\": container with ID starting with e732d6e90cba3c1879c10911b3e8d2ae65a20f52d586eec5fea922c9221bb064 not found: ID does not exist"
Nov 01 20:10:42.016238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.016166    5041 scope.go:122] "RemoveContainer" containerID="d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"
Nov 01 20:10:42.016472 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:10:42.016452    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d\": container with ID starting with d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d not found: ID does not exist" containerID="d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"
Nov 01 20:10:42.016520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.016478    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d"} err="failed to get container status \"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d\": rpc error: code = NotFound desc = could not find container \"d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d\": container with ID starting with d151b3b8a312aa9fed1b843afbbdbaa1fcf29ad3deaa470a1abae69dc038665d not found: ID does not exist"
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071520    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071619    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071727    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071797    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.071921    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.072052    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.072093    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\") pod \"15527bc2-40d5-4af4-9eb0-332391978340\" (UID: \"15527bc2-40d5-4af4-9eb0-332391978340\") "
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.079392    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.079806    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.080091    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:42.080188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.080157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:42.090534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.090467    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:42.102439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.102372    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:10:42.148531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.148408    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:42.156814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.149126    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:10:42.167869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.157638    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk" pod "15527bc2-40d5-4af4-9eb0-332391978340" (UID: "15527bc2-40d5-4af4-9eb0-332391978340"). InnerVolumeSpecName "kube-api-access-27gwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179237    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179274    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179294    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27gwk\" (UniqueName: \"kubernetes.io/projected/15527bc2-40d5-4af4-9eb0-332391978340-kube-api-access-27gwk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179310    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179325    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15527bc2-40d5-4af4-9eb0-332391978340-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179339    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179353    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15527bc2-40d5-4af4-9eb0-332391978340-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179366    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.194540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.179380    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15527bc2-40d5-4af4-9eb0-332391978340-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.301937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.301900    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:42.318230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.318193    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:10:42.327577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.327536    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4a127b-openshift-default-6bf94b6c47-7d9vx"]
Nov 01 20:10:42.431622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.431548    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15527bc2-40d5-4af4-9eb0-332391978340" path="/var/lib/kubelet/pods/15527bc2-40d5-4af4-9eb0-332391978340/volumes"
Nov 01 20:10:42.483706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.483672    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\") pod \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") "
Nov 01 20:10:42.483884 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.483762    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\") pod \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") "
Nov 01 20:10:42.483961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.483944    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\" (UniqueName: \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\") pod \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\" (UID: \"a1a7cbca-51b4-48a5-9d4e-1991b98068ad\") "
Nov 01 20:10:42.484174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.484146    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca" pod "a1a7cbca-51b4-48a5-9d4e-1991b98068ad" (UID: "a1a7cbca-51b4-48a5-9d4e-1991b98068ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:42.484430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.484406    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.485909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.485883    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca" pod "a1a7cbca-51b4-48a5-9d4e-1991b98068ad" (UID: "a1a7cbca-51b4-48a5-9d4e-1991b98068ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:42.486034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.486012    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg" pod "a1a7cbca-51b4-48a5-9d4e-1991b98068ad" (UID: "a1a7cbca-51b4-48a5-9d4e-1991b98068ad"). InnerVolumeSpecName "kube-api-access-f84qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:42.588802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.587953    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:42.588802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:42.587998    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84qg\" (UniqueName: \"kubernetes.io/projected/a1a7cbca-51b4-48a5-9d4e-1991b98068ad-kube-api-access-f84qg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:43.115804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:43.111124    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs" event={"ID":"a1a7cbca-51b4-48a5-9d4e-1991b98068ad","Type":"ContainerDied","Data":"cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8"}
Nov 01 20:10:43.115804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:43.111230    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdea2aa39077e5983857c02cd5dba3f599a2c0990dc566acb09ca4302cdad7c8"
Nov 01 20:10:43.115804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:43.111304    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"
Nov 01 20:10:44.598806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:44.586949    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.500804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:45.490108    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.650800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:45.649878    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:45.703800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:45.703098    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:45.735805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:45.704379    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:10:46.527674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.526961    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:10:46.671919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.671874    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:46.704934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.704887    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:10:46.734147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.734114    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:10:46.734709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.734689    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:10:46.807881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:46.807851    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:10:47.246879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:47.232850    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:10:47.676592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:47.676537    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:47.731833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:47.704117    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:10:48.889718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.889682    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"]
Nov 01 20:10:48.890516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890493    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a1a7cbca-51b4-48a5-9d4e-1991b98068ad" containerName="revoke-keys"
Nov 01 20:10:48.890618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890517    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a7cbca-51b4-48a5-9d4e-1991b98068ad" containerName="revoke-keys"
Nov 01 20:10:48.890618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890548    5041 cpu_manager.go:414] "containerMap: removing container" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:48.890618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890557    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:48.890820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890759    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:48.890820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890772    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:48.890820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.890808    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a7cbca-51b4-48a5-9d4e-1991b98068ad" containerName="revoke-keys"
Nov 01 20:10:48.895088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.895063    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:48.897999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.897975    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"]
Nov 01 20:10:48.898446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.898404    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:48.898446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.898413    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:48.901412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.901389    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:48.901537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:48.901522    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:49.031275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.031243    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.031451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.031366    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxh5\" (UniqueName: \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.031515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.031467    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.132752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.132714    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.132965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.132829    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.132965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.132931    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxh5\" (UniqueName: \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.133471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.133450    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.135285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.135264    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.140681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.140631    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxh5\" (UniqueName: \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\") pod \"maas-api-revoke-sub-0ae47fd2-grd9b\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.211540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.211505    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:49.258970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.258929    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"]
Nov 01 20:10:49.260670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.260640    5041 cpu_manager.go:414] "containerMap: removing container" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:49.261057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.261035    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15527bc2-40d5-4af4-9eb0-332391978340" containerName="istio-proxy"
Nov 01 20:10:49.266712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.266688    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.267333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.267305    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"]
Nov 01 20:10:49.437428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.437347    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxv8g\" (UniqueName: \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.437589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.437563    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.437726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.437703    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.502591 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:10:49.502544    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77494a1d_0d8d_4c26_a375_962e712d5717.slice/crio-484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf WatchSource:0}: Error finding container 484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf: Status 404 returned error can't find the container with id 484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf
Nov 01 20:10:49.504917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.504828    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"]
Nov 01 20:10:49.540848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.540270    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.540848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.540479    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.540848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.540632    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxv8g\" (UniqueName: \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.543657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.543629    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.546666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.546638    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.551044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.551014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxv8g\" (UniqueName: \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\") pod \"maas-api-revoke-sub-b915099b-f567n\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.588436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.588403    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:49.772341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.772306    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b" event={"ID":"77494a1d-0d8d-4c26-a375-962e712d5717","Type":"ContainerStarted","Data":"484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf"}
Nov 01 20:10:49.872236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:49.872204    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"]
Nov 01 20:10:49.872372 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:10:49.872321    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36e8776_861e_45d7_9f88_0a61c3a4eadb.slice/crio-1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b WatchSource:0}: Error finding container 1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b: Status 404 returned error can't find the container with id 1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b
Nov 01 20:10:50.783922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:50.783827    5041 generic.go:356] "Generic (PLEG): container finished" podID="77494a1d-0d8d-4c26-a375-962e712d5717" containerID="cda4f9604b80a5f5cb54d3a276602f785ec69cc497238d3fb3d6e5b531abf6cf" exitCode=0
Nov 01 20:10:50.783922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:50.783911    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b" event={"ID":"77494a1d-0d8d-4c26-a375-962e712d5717","Type":"ContainerDied","Data":"cda4f9604b80a5f5cb54d3a276602f785ec69cc497238d3fb3d6e5b531abf6cf"}
Nov 01 20:10:50.785795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:50.785754    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n" event={"ID":"d36e8776-861e-45d7-9f88-0a61c3a4eadb","Type":"ContainerStarted","Data":"cb49e776215f8bcd740b043c0f5f8500a65d067f808ffff79534ebc846ca009a"}
Nov 01 20:10:50.785915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:50.785826    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n" event={"ID":"d36e8776-861e-45d7-9f88-0a61c3a4eadb","Type":"ContainerStarted","Data":"1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b"}
Nov 01 20:10:50.813311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:50.813254    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n" podStartSLOduration=1.8132388160000001 podStartE2EDuration="1.813238816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:50.812798293 +0000 UTC m=+738.533876986" watchObservedRunningTime="2026-11-01 20:10:50.813238816 +0000 UTC m=+738.534317529"
Nov 01 20:10:51.794215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:51.794178    5041 generic.go:356] "Generic (PLEG): container finished" podID="d36e8776-861e-45d7-9f88-0a61c3a4eadb" containerID="cb49e776215f8bcd740b043c0f5f8500a65d067f808ffff79534ebc846ca009a" exitCode=0
Nov 01 20:10:51.794692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:51.794262    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n" event={"ID":"d36e8776-861e-45d7-9f88-0a61c3a4eadb","Type":"ContainerDied","Data":"cb49e776215f8bcd740b043c0f5f8500a65d067f808ffff79534ebc846ca009a"}
Nov 01 20:10:51.973042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:51.973011    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:52.079621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.079534    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\") pod \"77494a1d-0d8d-4c26-a375-962e712d5717\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") "
Nov 01 20:10:52.079820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.079637    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\" (UniqueName: \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\") pod \"77494a1d-0d8d-4c26-a375-962e712d5717\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") "
Nov 01 20:10:52.079820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.079671    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\") pod \"77494a1d-0d8d-4c26-a375-962e712d5717\" (UID: \"77494a1d-0d8d-4c26-a375-962e712d5717\") "
Nov 01 20:10:52.080007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.079982    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca" pod "77494a1d-0d8d-4c26-a375-962e712d5717" (UID: "77494a1d-0d8d-4c26-a375-962e712d5717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:52.080137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.080120    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:52.081862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.081829    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca" pod "77494a1d-0d8d-4c26-a375-962e712d5717" (UID: "77494a1d-0d8d-4c26-a375-962e712d5717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:52.081963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.081920    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5" pod "77494a1d-0d8d-4c26-a375-962e712d5717" (UID: "77494a1d-0d8d-4c26-a375-962e712d5717"). InnerVolumeSpecName "kube-api-access-dwxh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:52.181250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.181212    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxh5\" (UniqueName: \"kubernetes.io/projected/77494a1d-0d8d-4c26-a375-962e712d5717-kube-api-access-dwxh5\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:52.181250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.181246    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77494a1d-0d8d-4c26-a375-962e712d5717-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:52.804774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.804738    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b" event={"ID":"77494a1d-0d8d-4c26-a375-962e712d5717","Type":"ContainerDied","Data":"484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf"}
Nov 01 20:10:52.804774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.804770    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"
Nov 01 20:10:52.805251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:52.804814    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484c20f1a2f803764e3722712684a2e1c9c6630af3643be640bf393382e80fcf"
Nov 01 20:10:53.003714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.003686    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:53.098458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.098358    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\") pod \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") "
Nov 01 20:10:53.098658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.098489    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\") pod \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") "
Nov 01 20:10:53.098658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.098589    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\" (UniqueName: \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\") pod \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\" (UID: \"d36e8776-861e-45d7-9f88-0a61c3a4eadb\") "
Nov 01 20:10:53.098879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.098848    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca" pod "d36e8776-861e-45d7-9f88-0a61c3a4eadb" (UID: "d36e8776-861e-45d7-9f88-0a61c3a4eadb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:10:53.099075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.099056    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:53.100542 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.100517    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca" pod "d36e8776-861e-45d7-9f88-0a61c3a4eadb" (UID: "d36e8776-861e-45d7-9f88-0a61c3a4eadb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:10:53.100824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.100764    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g" pod "d36e8776-861e-45d7-9f88-0a61c3a4eadb" (UID: "d36e8776-861e-45d7-9f88-0a61c3a4eadb"). InnerVolumeSpecName "kube-api-access-gxv8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:10:53.200423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.200385    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxv8g\" (UniqueName: \"kubernetes.io/projected/d36e8776-861e-45d7-9f88-0a61c3a4eadb-kube-api-access-gxv8g\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:53.200423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.200417    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36e8776-861e-45d7-9f88-0a61c3a4eadb-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:10:53.814029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.814003    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"
Nov 01 20:10:53.814029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.814011    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n" event={"ID":"d36e8776-861e-45d7-9f88-0a61c3a4eadb","Type":"ContainerDied","Data":"1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b"}
Nov 01 20:10:53.814517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:53.814069    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d324a015c464b541071ea3e1ef8ef937abddfb0b2d80c16cc6c4ecfb741808b"
Nov 01 20:10:58.692804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:58.692705    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:58.812803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:58.811949    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:59.254892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.254836    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:10:59.255554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.255125    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:10:59.321136 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:10:59.321094    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:10:59.608803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.601059    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"]
Nov 01 20:10:59.644804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.613718    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d36e8776-861e-45d7-9f88-0a61c3a4eadb" containerName="revoke-keys"
Nov 01 20:10:59.648802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.645019    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36e8776-861e-45d7-9f88-0a61c3a4eadb" containerName="revoke-keys"
Nov 01 20:10:59.659801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.652861    5041 cpu_manager.go:414] "containerMap: removing container" podUID="77494a1d-0d8d-4c26-a375-962e712d5717" containerName="revoke-keys"
Nov 01 20:10:59.660006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.659981    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77494a1d-0d8d-4c26-a375-962e712d5717" containerName="revoke-keys"
Nov 01 20:10:59.688803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.664133    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="77494a1d-0d8d-4c26-a375-962e712d5717" containerName="revoke-keys"
Nov 01 20:10:59.689075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.689049    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36e8776-861e-45d7-9f88-0a61c3a4eadb" containerName="revoke-keys"
Nov 01 20:10:59.715804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.707056    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:10:59.765895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.764282    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"]
Nov 01 20:10:59.765895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.764378    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:10:59.765895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.765285    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:10:59.776332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.774966    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:10:59.776332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.776112    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:10:59.776332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.776204    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:10:59.776332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.776296    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:10:59.941238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.941135    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwjq\" (UniqueName: \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:10:59.941615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.941574    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:10:59.942550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:10:59.942522    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.092443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.075267    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.092443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.075373    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwjq\" (UniqueName: \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.092443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.075426    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.092443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.076436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.282855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.267629    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:11:00.282855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.268706    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:00.554801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.553577    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwjq\" (UniqueName: \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.633773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.633705    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c5b5b-lm5lj\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.765804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.716272    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:11:00.792673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.792385    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:00.815049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:00.814952    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:01.259681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:01.258631    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:11:01.321704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:01.321181    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:01.422177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:01.420491    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:01.694817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:01.694743    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"]
Nov 01 20:11:01.695410 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:01.695366    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4f1503_9f5b_44be_8ad3_325118ba45d3.slice/crio-db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2 WatchSource:0}: Error finding container db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2: Status 404 returned error can't find the container with id db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2
Nov 01 20:11:02.336237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:02.336197    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj" event={"ID":"ac4f1503-9f5b-44be-8ad3-325118ba45d3","Type":"ContainerStarted","Data":"aa87ca4c9ac46ae1da9384c4262ffabb47d4fac343edab5d9f498f29c3587b55"}
Nov 01 20:11:02.336629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:02.336257    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj" event={"ID":"ac4f1503-9f5b-44be-8ad3-325118ba45d3","Type":"ContainerStarted","Data":"db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2"}
Nov 01 20:11:03.345668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:03.345633    5041 generic.go:356] "Generic (PLEG): container finished" podID="ac4f1503-9f5b-44be-8ad3-325118ba45d3" containerID="aa87ca4c9ac46ae1da9384c4262ffabb47d4fac343edab5d9f498f29c3587b55" exitCode=0
Nov 01 20:11:03.346070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:03.345710    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj" event={"ID":"ac4f1503-9f5b-44be-8ad3-325118ba45d3","Type":"ContainerDied","Data":"aa87ca4c9ac46ae1da9384c4262ffabb47d4fac343edab5d9f498f29c3587b55"}
Nov 01 20:11:04.056736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.056689    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"]
Nov 01 20:11:04.061324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.061302    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.078655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.078536    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"]
Nov 01 20:11:04.106389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.106354    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.106768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.106562    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.106768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.106612    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.207696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.207656    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.207932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.207707    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.207932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.207847    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.208470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.208444    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.210131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.210101    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.215553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.215530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\") pod \"maas-api-revoke-sub-f7dd1877-nw7lf\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.368384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.368347    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"]
Nov 01 20:11:04.373383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.373358    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.399197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.387892    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:04.399197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.393909    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"]
Nov 01 20:11:04.421297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.421262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbnc\" (UniqueName: \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.422999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.422972    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.423087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.423052    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.563444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.525580    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbnc\" (UniqueName: \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.563444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.525699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.563444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.525797    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.563444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.554412    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.565880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.565836    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.582997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.582603    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbnc\" (UniqueName: \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\") pod \"maas-api-revoke-sub-3b07fe65-mzj5w\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.611841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.607026    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:04.702390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.702361    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:04.718261 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:04.718231    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f548acd_ed33_4e54_a418_59f5bb610eaa.slice/crio-8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262 WatchSource:0}: Error finding container 8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262: Status 404 returned error can't find the container with id 8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262
Nov 01 20:11:04.718523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.718505    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"]
Nov 01 20:11:04.731745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.731720    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\") pod \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") "
Nov 01 20:11:04.731905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.731770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\" (UniqueName: \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\") pod \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") "
Nov 01 20:11:04.731905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.731826    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\") pod \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\" (UID: \"ac4f1503-9f5b-44be-8ad3-325118ba45d3\") "
Nov 01 20:11:04.732307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.732282    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca" pod "ac4f1503-9f5b-44be-8ad3-325118ba45d3" (UID: "ac4f1503-9f5b-44be-8ad3-325118ba45d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:04.732558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.732515    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:04.734048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.734024    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca" pod "ac4f1503-9f5b-44be-8ad3-325118ba45d3" (UID: "ac4f1503-9f5b-44be-8ad3-325118ba45d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:04.734596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.734567    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq" pod "ac4f1503-9f5b-44be-8ad3-325118ba45d3" (UID: "ac4f1503-9f5b-44be-8ad3-325118ba45d3"). InnerVolumeSpecName "kube-api-access-ggwjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:04.834405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.834369    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggwjq\" (UniqueName: \"kubernetes.io/projected/ac4f1503-9f5b-44be-8ad3-325118ba45d3-kube-api-access-ggwjq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:04.834405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:04.834407    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4f1503-9f5b-44be-8ad3-325118ba45d3-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:05.011840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.011713    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"]
Nov 01 20:11:05.019488 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:05.019460    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea89c80_951d_40f0_b059_7475a699d1ae.slice/crio-daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42 WatchSource:0}: Error finding container daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42: Status 404 returned error can't find the container with id daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42
Nov 01 20:11:05.365031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.364993    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w" event={"ID":"eea89c80-951d-40f0-b059-7475a699d1ae","Type":"ContainerStarted","Data":"daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42"}
Nov 01 20:11:05.366385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.366362    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"
Nov 01 20:11:05.366385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.366371    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj" event={"ID":"ac4f1503-9f5b-44be-8ad3-325118ba45d3","Type":"ContainerDied","Data":"db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2"}
Nov 01 20:11:05.366562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.366416    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db15587f6d9c991c4c6c2c62efaabe638121d61d6c3411575ed8f62972800af2"
Nov 01 20:11:05.368116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.368090    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf" event={"ID":"4f548acd-ed33-4e54-a418-59f5bb610eaa","Type":"ContainerStarted","Data":"417b9f560106ce31b6917aef42cbcfcd23dc23724b6c6044da9bc9e0f0896e47"}
Nov 01 20:11:05.368226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.368140    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf" event={"ID":"4f548acd-ed33-4e54-a418-59f5bb610eaa","Type":"ContainerStarted","Data":"8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262"}
Nov 01 20:11:05.387198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:05.387151    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf" podStartSLOduration=1.387137306 podStartE2EDuration="1.387137306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:05.383454218 +0000 UTC m=+753.104532896" watchObservedRunningTime="2026-11-01 20:11:05.387137306 +0000 UTC m=+753.108216007"
Nov 01 20:11:06.382653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:06.382615    5041 generic.go:356] "Generic (PLEG): container finished" podID="4f548acd-ed33-4e54-a418-59f5bb610eaa" containerID="417b9f560106ce31b6917aef42cbcfcd23dc23724b6c6044da9bc9e0f0896e47" exitCode=0
Nov 01 20:11:06.382830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:06.382696    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf" event={"ID":"4f548acd-ed33-4e54-a418-59f5bb610eaa","Type":"ContainerDied","Data":"417b9f560106ce31b6917aef42cbcfcd23dc23724b6c6044da9bc9e0f0896e47"}
Nov 01 20:11:06.384647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:06.384620    5041 generic.go:356] "Generic (PLEG): container finished" podID="eea89c80-951d-40f0-b059-7475a699d1ae" containerID="18f3ee07085ff0fad61792ab90bfb4bb93942b83f90a31741a6876fc6e9f949f" exitCode=0
Nov 01 20:11:06.384751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:06.384690    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w" event={"ID":"eea89c80-951d-40f0-b059-7475a699d1ae","Type":"ContainerDied","Data":"18f3ee07085ff0fad61792ab90bfb4bb93942b83f90a31741a6876fc6e9f949f"}
Nov 01 20:11:07.583496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.583460    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:07.648761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.648691    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:07.675305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.675265    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\" (UniqueName: \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\") pod \"eea89c80-951d-40f0-b059-7475a699d1ae\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") "
Nov 01 20:11:07.675544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.675378    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\") pod \"4f548acd-ed33-4e54-a418-59f5bb610eaa\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") "
Nov 01 20:11:07.675544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.675441    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\") pod \"eea89c80-951d-40f0-b059-7475a699d1ae\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") "
Nov 01 20:11:07.676208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.676187    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\") pod \"4f548acd-ed33-4e54-a418-59f5bb610eaa\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") "
Nov 01 20:11:07.677171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.676727    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca" pod "4f548acd-ed33-4e54-a418-59f5bb610eaa" (UID: "4f548acd-ed33-4e54-a418-59f5bb610eaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:07.677317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.677300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\") pod \"eea89c80-951d-40f0-b059-7475a699d1ae\" (UID: \"eea89c80-951d-40f0-b059-7475a699d1ae\") "
Nov 01 20:11:07.677920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.677744    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca" pod "eea89c80-951d-40f0-b059-7475a699d1ae" (UID: "eea89c80-951d-40f0-b059-7475a699d1ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:07.678432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.678403    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca" pod "eea89c80-951d-40f0-b059-7475a699d1ae" (UID: "eea89c80-951d-40f0-b059-7475a699d1ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:07.678545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.678141    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\") pod \"4f548acd-ed33-4e54-a418-59f5bb610eaa\" (UID: \"4f548acd-ed33-4e54-a418-59f5bb610eaa\") "
Nov 01 20:11:07.679163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.679142    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:07.679238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.679171    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:07.679238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.679186    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eea89c80-951d-40f0-b059-7475a699d1ae-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:07.681863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.681827    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc" pod "eea89c80-951d-40f0-b059-7475a699d1ae" (UID: "eea89c80-951d-40f0-b059-7475a699d1ae"). InnerVolumeSpecName "kube-api-access-rjbnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:07.682172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.682141    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca" pod "4f548acd-ed33-4e54-a418-59f5bb610eaa" (UID: "4f548acd-ed33-4e54-a418-59f5bb610eaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:07.683807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.683767    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2" pod "4f548acd-ed33-4e54-a418-59f5bb610eaa" (UID: "4f548acd-ed33-4e54-a418-59f5bb610eaa"). InnerVolumeSpecName "kube-api-access-w2pf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:07.780260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.780225    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2pf2\" (UniqueName: \"kubernetes.io/projected/4f548acd-ed33-4e54-a418-59f5bb610eaa-kube-api-access-w2pf2\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:07.780260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.780258    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbnc\" (UniqueName: \"kubernetes.io/projected/eea89c80-951d-40f0-b059-7475a699d1ae-kube-api-access-rjbnc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:07.780481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:07.780272    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f548acd-ed33-4e54-a418-59f5bb610eaa-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:08.404333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.404296    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"
Nov 01 20:11:08.404535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.404330    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w" event={"ID":"eea89c80-951d-40f0-b059-7475a699d1ae","Type":"ContainerDied","Data":"daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42"}
Nov 01 20:11:08.404535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.404424    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daec707786f49b48592ddfc31851bbd2abe1d84132dca3ff2a476d69432f0e42"
Nov 01 20:11:08.409900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.409871    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"
Nov 01 20:11:08.410072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.409871    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf" event={"ID":"4f548acd-ed33-4e54-a418-59f5bb610eaa","Type":"ContainerDied","Data":"8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262"}
Nov 01 20:11:08.410072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:08.409998    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d767c2ce7b42df23e03286caf575c286cfa6f57997786724fe670a0e958a262"
Nov 01 20:11:14.811987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:14.811871    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.334362    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"]
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.335545    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4f548acd-ed33-4e54-a418-59f5bb610eaa" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.335571    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f548acd-ed33-4e54-a418-59f5bb610eaa" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336265    5041 cpu_manager.go:414] "containerMap: removing container" podUID="eea89c80-951d-40f0-b059-7475a699d1ae" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336295    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea89c80-951d-40f0-b059-7475a699d1ae" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336318    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ac4f1503-9f5b-44be-8ad3-325118ba45d3" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336326    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4f1503-9f5b-44be-8ad3-325118ba45d3" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336513    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea89c80-951d-40f0-b059-7475a699d1ae" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336526    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4f1503-9f5b-44be-8ad3-325118ba45d3" containerName="revoke-keys"
Nov 01 20:11:15.336803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.336548    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f548acd-ed33-4e54-a418-59f5bb610eaa" containerName="revoke-keys"
Nov 01 20:11:15.368804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.368466    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"]
Nov 01 20:11:15.368804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.368614    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.467166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.467135    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.477561    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tllv\" (UniqueName: \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.490059    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.490158    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.478095    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.477875    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:11:15.492802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.477982    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:11:15.601177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.601083    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tllv\" (UniqueName: \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.601705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.601683    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.601973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.601955    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.605075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.604097    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.626123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.626079    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.688325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.687505    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tllv\" (UniqueName: \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\") pod \"maas-api-revoke-sub-1f9c938c-6qhq7\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:15.780323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:15.779251    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:16.210445 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:16.210410    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4793189_93dc_4630_a127_bd5d9ea5fd45.slice/crio-ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361 WatchSource:0}: Error finding container ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361: Status 404 returned error can't find the container with id ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361
Nov 01 20:11:16.211774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.211744    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"]
Nov 01 20:11:16.697007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.696975    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"]
Nov 01 20:11:16.703485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.703459    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.711896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.711821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7" event={"ID":"c4793189-93dc-4630-a127-bd5d9ea5fd45","Type":"ContainerStarted","Data":"fde913bfaa275b2062d72f7ab8833b82d2f1df9dca495dd767f12a4607878ddf"}
Nov 01 20:11:16.712085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.712064    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7" event={"ID":"c4793189-93dc-4630-a127-bd5d9ea5fd45","Type":"ContainerStarted","Data":"ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361"}
Nov 01 20:11:16.716705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.716680    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"]
Nov 01 20:11:16.725097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.725068    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.725305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.725285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btr2\" (UniqueName: \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.725438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.725420    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.737827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.737721    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7" podStartSLOduration=1.737702599 podStartE2EDuration="1.737702599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:16.735064337 +0000 UTC m=+764.456143030" watchObservedRunningTime="2026-11-01 20:11:16.737702599 +0000 UTC m=+764.458781276"
Nov 01 20:11:16.826870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.826828    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.827148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.827123    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.827220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.827195    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btr2\" (UniqueName: \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.827842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.827819    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.829340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.829321    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:16.834508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:16.834477    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btr2\" (UniqueName: \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\") pod \"maas-api-revoke-sub-9122cc42-259ll\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:17.024203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:17.024119    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:17.308598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:17.308570    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"]
Nov 01 20:11:17.325848 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:17.325817    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0aa61_11b3_405f_a19c_c51e9c616272.slice/crio-6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53 WatchSource:0}: Error finding container 6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53: Status 404 returned error can't find the container with id 6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53
Nov 01 20:11:17.722262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:17.722227    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll" event={"ID":"b5a0aa61-11b3-405f-a19c-c51e9c616272","Type":"ContainerStarted","Data":"6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53"}
Nov 01 20:11:17.723927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:17.723902    5041 generic.go:356] "Generic (PLEG): container finished" podID="c4793189-93dc-4630-a127-bd5d9ea5fd45" containerID="fde913bfaa275b2062d72f7ab8833b82d2f1df9dca495dd767f12a4607878ddf" exitCode=0
Nov 01 20:11:17.724060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:17.723938    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7" event={"ID":"c4793189-93dc-4630-a127-bd5d9ea5fd45","Type":"ContainerDied","Data":"fde913bfaa275b2062d72f7ab8833b82d2f1df9dca495dd767f12a4607878ddf"}
Nov 01 20:11:18.733918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.733835    5041 generic.go:356] "Generic (PLEG): container finished" podID="b5a0aa61-11b3-405f-a19c-c51e9c616272" containerID="6b3d745b09ee05316ff7def291f9f81a1932ceb99edd1751728df8c1ca427161" exitCode=0
Nov 01 20:11:18.733918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.733898    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll" event={"ID":"b5a0aa61-11b3-405f-a19c-c51e9c616272","Type":"ContainerDied","Data":"6b3d745b09ee05316ff7def291f9f81a1932ceb99edd1751728df8c1ca427161"}
Nov 01 20:11:18.897734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.897708    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:18.954342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.954300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\") pod \"c4793189-93dc-4630-a127-bd5d9ea5fd45\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") "
Nov 01 20:11:18.954524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.954400    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\") pod \"c4793189-93dc-4630-a127-bd5d9ea5fd45\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") "
Nov 01 20:11:18.954524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.954489    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\" (UniqueName: \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\") pod \"c4793189-93dc-4630-a127-bd5d9ea5fd45\" (UID: \"c4793189-93dc-4630-a127-bd5d9ea5fd45\") "
Nov 01 20:11:18.955401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.954817    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca" pod "c4793189-93dc-4630-a127-bd5d9ea5fd45" (UID: "c4793189-93dc-4630-a127-bd5d9ea5fd45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:18.955401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.955124    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:18.957136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.957107    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca" pod "c4793189-93dc-4630-a127-bd5d9ea5fd45" (UID: "c4793189-93dc-4630-a127-bd5d9ea5fd45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:18.957250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.957217    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv" pod "c4793189-93dc-4630-a127-bd5d9ea5fd45" (UID: "c4793189-93dc-4630-a127-bd5d9ea5fd45"). InnerVolumeSpecName "kube-api-access-6tllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:18.995277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.995204    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"]
Nov 01 20:11:18.995858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.995846    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c4793189-93dc-4630-a127-bd5d9ea5fd45" containerName="revoke-keys"
Nov 01 20:11:18.995909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.995860    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4793189-93dc-4630-a127-bd5d9ea5fd45" containerName="revoke-keys"
Nov 01 20:11:18.995962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.995953    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4793189-93dc-4630-a127-bd5d9ea5fd45" containerName="revoke-keys"
Nov 01 20:11:19.000207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:18.999895    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.007643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.007618    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"]
Nov 01 20:11:19.056257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.056224    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg46m\" (UniqueName: \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.056439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.056408    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.056488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.056458    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.056659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.056644    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4793189-93dc-4630-a127-bd5d9ea5fd45-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:19.056698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.056666    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tllv\" (UniqueName: \"kubernetes.io/projected/c4793189-93dc-4630-a127-bd5d9ea5fd45-kube-api-access-6tllv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:19.157901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.157870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.158106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.157908    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.158184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.158103    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg46m\" (UniqueName: \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.158587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.158564    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.160584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.160557    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.169025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.169001    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg46m\" (UniqueName: \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\") pod \"maas-api-revoke-sub-481c79e3-bmtlw\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.322342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.322311    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:19.602180 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:19.602145    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c51f4_5256_48d6_999d_c3aef4a2e710.slice/crio-babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977 WatchSource:0}: Error finding container babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977: Status 404 returned error can't find the container with id babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977
Nov 01 20:11:19.603660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.603635    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"]
Nov 01 20:11:19.743608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.743577    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"
Nov 01 20:11:19.744068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.743614    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7" event={"ID":"c4793189-93dc-4630-a127-bd5d9ea5fd45","Type":"ContainerDied","Data":"ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361"}
Nov 01 20:11:19.744068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.743670    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebfa66378f3b28c1d44a9aaa0f918a3508f338c3664417fb264c4d264b638361"
Nov 01 20:11:19.746027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.745999    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw" event={"ID":"1a3c51f4-5256-48d6-999d-c3aef4a2e710","Type":"ContainerStarted","Data":"babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977"}
Nov 01 20:11:19.919212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.919181    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:19.967790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.967751    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\" (UniqueName: \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\") pod \"b5a0aa61-11b3-405f-a19c-c51e9c616272\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") "
Nov 01 20:11:19.967930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.967834    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\") pod \"b5a0aa61-11b3-405f-a19c-c51e9c616272\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") "
Nov 01 20:11:19.967974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.967949    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\") pod \"b5a0aa61-11b3-405f-a19c-c51e9c616272\" (UID: \"b5a0aa61-11b3-405f-a19c-c51e9c616272\") "
Nov 01 20:11:19.968296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.968274    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca" pod "b5a0aa61-11b3-405f-a19c-c51e9c616272" (UID: "b5a0aa61-11b3-405f-a19c-c51e9c616272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:19.968437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.968422    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:19.970045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.970021    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca" pod "b5a0aa61-11b3-405f-a19c-c51e9c616272" (UID: "b5a0aa61-11b3-405f-a19c-c51e9c616272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:19.970137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:19.970043    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2" pod "b5a0aa61-11b3-405f-a19c-c51e9c616272" (UID: "b5a0aa61-11b3-405f-a19c-c51e9c616272"). InnerVolumeSpecName "kube-api-access-2btr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:20.069238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.069202    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btr2\" (UniqueName: \"kubernetes.io/projected/b5a0aa61-11b3-405f-a19c-c51e9c616272-kube-api-access-2btr2\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:20.069238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.069232    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0aa61-11b3-405f-a19c-c51e9c616272-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:20.756290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.756200    5041 generic.go:356] "Generic (PLEG): container finished" podID="1a3c51f4-5256-48d6-999d-c3aef4a2e710" containerID="1a65f1032d6f79c48ed6cd1a76768da9e591f5172bed14e9d3f49805b6c35574" exitCode=0
Nov 01 20:11:20.756290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.756242    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw" event={"ID":"1a3c51f4-5256-48d6-999d-c3aef4a2e710","Type":"ContainerDied","Data":"1a65f1032d6f79c48ed6cd1a76768da9e591f5172bed14e9d3f49805b6c35574"}
Nov 01 20:11:20.758006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.757988    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"
Nov 01 20:11:20.758107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.757991    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll" event={"ID":"b5a0aa61-11b3-405f-a19c-c51e9c616272","Type":"ContainerDied","Data":"6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53"}
Nov 01 20:11:20.758107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:20.758096    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c778ab01d502667f565fd69ffe5134c053f6eb826784e48bfbe260d78201d53"
Nov 01 20:11:21.929312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:21.929284    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:22.017893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.017812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\") pod \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") "
Nov 01 20:11:22.018054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.017930    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\") pod \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") "
Nov 01 20:11:22.018054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.017968    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\" (UniqueName: \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\") pod \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\" (UID: \"1a3c51f4-5256-48d6-999d-c3aef4a2e710\") "
Nov 01 20:11:22.018255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.018212    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca" pod "1a3c51f4-5256-48d6-999d-c3aef4a2e710" (UID: "1a3c51f4-5256-48d6-999d-c3aef4a2e710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:22.018601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.018574    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:22.020028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.020001    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca" pod "1a3c51f4-5256-48d6-999d-c3aef4a2e710" (UID: "1a3c51f4-5256-48d6-999d-c3aef4a2e710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:22.020226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.020204    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m" pod "1a3c51f4-5256-48d6-999d-c3aef4a2e710" (UID: "1a3c51f4-5256-48d6-999d-c3aef4a2e710"). InnerVolumeSpecName "kube-api-access-dg46m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:22.119738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.119689    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a3c51f4-5256-48d6-999d-c3aef4a2e710-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:22.119738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.119739    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg46m\" (UniqueName: \"kubernetes.io/projected/1a3c51f4-5256-48d6-999d-c3aef4a2e710-kube-api-access-dg46m\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:22.782122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.782039    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"
Nov 01 20:11:22.782122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.782081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw" event={"ID":"1a3c51f4-5256-48d6-999d-c3aef4a2e710","Type":"ContainerDied","Data":"babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977"}
Nov 01 20:11:22.782346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:22.782129    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="babfe843eb7c314c05343a1a912dda457c0f7c1446dc8dff228743c74a280977"
Nov 01 20:11:29.110811 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:11:29.106966    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:11:29.252805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:29.249852    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:29.458862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:29.457975    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:29.853940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:29.853890    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:29.900355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:29.900313    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:11:31.544925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.544893    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"]
Nov 01 20:11:31.545491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545477    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1a3c51f4-5256-48d6-999d-c3aef4a2e710" containerName="revoke-keys"
Nov 01 20:11:31.545491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545491    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3c51f4-5256-48d6-999d-c3aef4a2e710" containerName="revoke-keys"
Nov 01 20:11:31.545626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545534    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b5a0aa61-11b3-405f-a19c-c51e9c616272" containerName="revoke-keys"
Nov 01 20:11:31.545626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545539    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a0aa61-11b3-405f-a19c-c51e9c616272" containerName="revoke-keys"
Nov 01 20:11:31.545734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545635    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a0aa61-11b3-405f-a19c-c51e9c616272" containerName="revoke-keys"
Nov 01 20:11:31.545734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.545646    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3c51f4-5256-48d6-999d-c3aef4a2e710" containerName="revoke-keys"
Nov 01 20:11:31.549875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.549853    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.554029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.554008    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:11:31.554191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.554084    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:11:31.554966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.554947    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:11:31.555116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.554992    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:11:31.557078 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.557059    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"]
Nov 01 20:11:31.569883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.569850    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.569966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.569924    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.570032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.570017    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvnp\" (UniqueName: \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.672201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.672135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.672384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.672242    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.672384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.672332    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvnp\" (UniqueName: \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.672953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.672923    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.675245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.675205    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.680889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.680868    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvnp\" (UniqueName: \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\") pod \"maas-api-revoke-sub-2728798a-fk7vw\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:31.873761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:31.873673    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:32.157445 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:32.157407    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0affbdbf_051b_410f_aee1_1e2fc41643b2.slice/crio-c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee WatchSource:0}: Error finding container c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee: Status 404 returned error can't find the container with id c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee
Nov 01 20:11:32.158266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:32.158151    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"]
Nov 01 20:11:32.303085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:32.303051    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw" event={"ID":"0affbdbf-051b-410f-aee1-1e2fc41643b2","Type":"ContainerStarted","Data":"c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee"}
Nov 01 20:11:33.009322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.009288    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"]
Nov 01 20:11:33.016105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.016071    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.023131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.023097    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"]
Nov 01 20:11:33.092125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.092088    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.092331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.092310    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.092390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.092376    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mktwd\" (UniqueName: \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.193269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.193241    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.193440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.193421    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.193526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.193510    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mktwd\" (UniqueName: \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.194120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.194094    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.195661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.195642    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.201077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.201054    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mktwd\" (UniqueName: \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\") pod \"maas-api-revoke-sub-22b2a433-b7rvp\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.313213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.313183    5041 generic.go:356] "Generic (PLEG): container finished" podID="0affbdbf-051b-410f-aee1-1e2fc41643b2" containerID="32adb2872416cf06c22ad5aaf6d4ce5a3c97e94afc051542b0af42bd19de5b84" exitCode=0
Nov 01 20:11:33.313390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.313267    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw" event={"ID":"0affbdbf-051b-410f-aee1-1e2fc41643b2","Type":"ContainerDied","Data":"32adb2872416cf06c22ad5aaf6d4ce5a3c97e94afc051542b0af42bd19de5b84"}
Nov 01 20:11:33.337791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.337761    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:33.620963 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:33.620931    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4822bd_1d0c_43d8_9ec0_7d133866815b.slice/crio-7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c WatchSource:0}: Error finding container 7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c: Status 404 returned error can't find the container with id 7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c
Nov 01 20:11:33.622086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:33.622056    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"]
Nov 01 20:11:34.349838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.345364    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp" event={"ID":"fe4822bd-1d0c-43d8-9ec0-7d133866815b","Type":"ContainerStarted","Data":"6c94cfd7a51f0ba58054c49318959e7b4379c99d8b701e43f0a4fddd1845a65d"}
Nov 01 20:11:34.349838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.345427    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp" event={"ID":"fe4822bd-1d0c-43d8-9ec0-7d133866815b","Type":"ContainerStarted","Data":"7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c"}
Nov 01 20:11:34.373181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.373112    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp" podStartSLOduration=2.37309066 podStartE2EDuration="2.37309066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:34.363157471 +0000 UTC m=+782.084236185" watchObservedRunningTime="2026-11-01 20:11:34.37309066 +0000 UTC m=+782.094169337"
Nov 01 20:11:34.627748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.627689    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:34.717435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.717403    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\") pod \"0affbdbf-051b-410f-aee1-1e2fc41643b2\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") "
Nov 01 20:11:34.717625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.717549    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\" (UniqueName: \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\") pod \"0affbdbf-051b-410f-aee1-1e2fc41643b2\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") "
Nov 01 20:11:34.717625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.717591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\") pod \"0affbdbf-051b-410f-aee1-1e2fc41643b2\" (UID: \"0affbdbf-051b-410f-aee1-1e2fc41643b2\") "
Nov 01 20:11:34.717840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.717817    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca" pod "0affbdbf-051b-410f-aee1-1e2fc41643b2" (UID: "0affbdbf-051b-410f-aee1-1e2fc41643b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:34.718058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.718039    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:34.719721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.719688    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp" pod "0affbdbf-051b-410f-aee1-1e2fc41643b2" (UID: "0affbdbf-051b-410f-aee1-1e2fc41643b2"). InnerVolumeSpecName "kube-api-access-qwvnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:34.719721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.719699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca" pod "0affbdbf-051b-410f-aee1-1e2fc41643b2" (UID: "0affbdbf-051b-410f-aee1-1e2fc41643b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:34.819503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.819473    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvnp\" (UniqueName: \"kubernetes.io/projected/0affbdbf-051b-410f-aee1-1e2fc41643b2-kube-api-access-qwvnp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:34.819503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.819501    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0affbdbf-051b-410f-aee1-1e2fc41643b2-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:34.899752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.899672    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"]
Nov 01 20:11:34.900390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.900372    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0affbdbf-051b-410f-aee1-1e2fc41643b2" containerName="revoke-keys"
Nov 01 20:11:34.900450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.900392    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0affbdbf-051b-410f-aee1-1e2fc41643b2" containerName="revoke-keys"
Nov 01 20:11:34.900581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.900569    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0affbdbf-051b-410f-aee1-1e2fc41643b2" containerName="revoke-keys"
Nov 01 20:11:34.904556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.904531    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:34.913191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:34.912498    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"]
Nov 01 20:11:35.022048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.022007    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcqr\" (UniqueName: \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.022210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.022130    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.022210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.022168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.123854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.123813    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcqr\" (UniqueName: \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.124021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.123915    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.124021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.123939    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.124657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.124637    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.126288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.126265    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.131569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.131547    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcqr\" (UniqueName: \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\") pod \"maas-api-revoke-sub-7674458c-s69z9\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.223614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.223535    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:35.357015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.356988    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"
Nov 01 20:11:35.357434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.357028    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw" event={"ID":"0affbdbf-051b-410f-aee1-1e2fc41643b2","Type":"ContainerDied","Data":"c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee"}
Nov 01 20:11:35.357434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.357087    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ff943a93dd931587f901ed3c327036a9ce2b081a2a4f641e4feb8e9d7c70ee"
Nov 01 20:11:35.359357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.359231    5041 generic.go:356] "Generic (PLEG): container finished" podID="fe4822bd-1d0c-43d8-9ec0-7d133866815b" containerID="6c94cfd7a51f0ba58054c49318959e7b4379c99d8b701e43f0a4fddd1845a65d" exitCode=0
Nov 01 20:11:35.359357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.359305    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp" event={"ID":"fe4822bd-1d0c-43d8-9ec0-7d133866815b","Type":"ContainerDied","Data":"6c94cfd7a51f0ba58054c49318959e7b4379c99d8b701e43f0a4fddd1845a65d"}
Nov 01 20:11:35.504288 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:11:35.504260    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9439cd2_366b_42c4_a049_e17117e6ea74.slice/crio-8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657 WatchSource:0}: Error finding container 8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657: Status 404 returned error can't find the container with id 8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657
Nov 01 20:11:35.505814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:35.505773    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"]
Nov 01 20:11:36.368494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.368457    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9" event={"ID":"a9439cd2-366b-42c4-a049-e17117e6ea74","Type":"ContainerStarted","Data":"f8862d0bed1b176565eaab662c0bdaa0104381b342c1e042cbc2e95a7b4ae13b"}
Nov 01 20:11:36.368948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.368520    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9" event={"ID":"a9439cd2-366b-42c4-a049-e17117e6ea74","Type":"ContainerStarted","Data":"8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657"}
Nov 01 20:11:36.389859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.389803    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9" podStartSLOduration=2.389770948 podStartE2EDuration="2.389770948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:36.384437816 +0000 UTC m=+784.105516492" watchObservedRunningTime="2026-11-01 20:11:36.389770948 +0000 UTC m=+784.110849628"
Nov 01 20:11:36.543311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.543285    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:36.643128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.642996    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\") pod \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") "
Nov 01 20:11:36.643283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.643246    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\" (UniqueName: \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\") pod \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") "
Nov 01 20:11:36.643361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.643291    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\") pod \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\" (UID: \"fe4822bd-1d0c-43d8-9ec0-7d133866815b\") "
Nov 01 20:11:36.643755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.643720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca" pod "fe4822bd-1d0c-43d8-9ec0-7d133866815b" (UID: "fe4822bd-1d0c-43d8-9ec0-7d133866815b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:36.645525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.645500    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca" pod "fe4822bd-1d0c-43d8-9ec0-7d133866815b" (UID: "fe4822bd-1d0c-43d8-9ec0-7d133866815b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:36.645608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.645588    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd" pod "fe4822bd-1d0c-43d8-9ec0-7d133866815b" (UID: "fe4822bd-1d0c-43d8-9ec0-7d133866815b"). InnerVolumeSpecName "kube-api-access-mktwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:36.745834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.745795    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:36.745834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.745833    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mktwd\" (UniqueName: \"kubernetes.io/projected/fe4822bd-1d0c-43d8-9ec0-7d133866815b-kube-api-access-mktwd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:36.745834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:36.745845    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe4822bd-1d0c-43d8-9ec0-7d133866815b-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:37.379521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:37.379489    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"
Nov 01 20:11:37.380006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:37.379524    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp" event={"ID":"fe4822bd-1d0c-43d8-9ec0-7d133866815b","Type":"ContainerDied","Data":"7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c"}
Nov 01 20:11:37.380006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:37.379602    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f61959d093d990cc25e3244a774836d660d3134ac4bd05950daf8e37da3200c"
Nov 01 20:11:37.381753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:37.381726    5041 generic.go:356] "Generic (PLEG): container finished" podID="a9439cd2-366b-42c4-a049-e17117e6ea74" containerID="f8862d0bed1b176565eaab662c0bdaa0104381b342c1e042cbc2e95a7b4ae13b" exitCode=0
Nov 01 20:11:37.381920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:37.381832    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9" event={"ID":"a9439cd2-366b-42c4-a049-e17117e6ea74","Type":"ContainerDied","Data":"f8862d0bed1b176565eaab662c0bdaa0104381b342c1e042cbc2e95a7b4ae13b"}
Nov 01 20:11:38.566868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.566838    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:38.669807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.669715    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\") pod \"a9439cd2-366b-42c4-a049-e17117e6ea74\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") "
Nov 01 20:11:38.669807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.669759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\") pod \"a9439cd2-366b-42c4-a049-e17117e6ea74\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") "
Nov 01 20:11:38.669995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.669887    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\" (UniqueName: \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\") pod \"a9439cd2-366b-42c4-a049-e17117e6ea74\" (UID: \"a9439cd2-366b-42c4-a049-e17117e6ea74\") "
Nov 01 20:11:38.670078 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.670056    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca" pod "a9439cd2-366b-42c4-a049-e17117e6ea74" (UID: "a9439cd2-366b-42c4-a049-e17117e6ea74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:38.670385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.670364    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:38.671993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.671966    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca" pod "a9439cd2-366b-42c4-a049-e17117e6ea74" (UID: "a9439cd2-366b-42c4-a049-e17117e6ea74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:38.671993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.671967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr" pod "a9439cd2-366b-42c4-a049-e17117e6ea74" (UID: "a9439cd2-366b-42c4-a049-e17117e6ea74"). InnerVolumeSpecName "kube-api-access-krcqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:38.771391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.771359    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcqr\" (UniqueName: \"kubernetes.io/projected/a9439cd2-366b-42c4-a049-e17117e6ea74-kube-api-access-krcqr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:38.771391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:38.771388    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9439cd2-366b-42c4-a049-e17117e6ea74-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:39.399118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:39.399088    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"
Nov 01 20:11:39.399118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:39.399105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9" event={"ID":"a9439cd2-366b-42c4-a049-e17117e6ea74","Type":"ContainerDied","Data":"8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657"}
Nov 01 20:11:39.399307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:39.399156    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8103e61495c35111753b6cd692f3ea6dc59cf584a4a7a44c43b0279b4839f657"
Nov 01 20:11:43.820804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.815938    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Nov 01 20:11:43.820804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.816567    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.996896    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"]
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.997905    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fe4822bd-1d0c-43d8-9ec0-7d133866815b" containerName="revoke-keys"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.997928    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4822bd-1d0c-43d8-9ec0-7d133866815b" containerName="revoke-keys"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.997947    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a9439cd2-366b-42c4-a049-e17117e6ea74" containerName="revoke-keys"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.997957    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9439cd2-366b-42c4-a049-e17117e6ea74" containerName="revoke-keys"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.998128    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4822bd-1d0c-43d8-9ec0-7d133866815b" containerName="revoke-keys"
Nov 01 20:11:44.000811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:43.998144    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9439cd2-366b-42c4-a049-e17117e6ea74" containerName="revoke-keys"
Nov 01 20:11:44.130856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.124094    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.172854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.160147    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:11:44.172854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.171183    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:11:44.183803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.181029    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:11:44.183803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.181158    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:11:44.207805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.204540    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"]
Nov 01 20:11:44.254803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.254695    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.254803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.254798    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c572\" (UniqueName: \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.255066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.254889    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.384860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.384594    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.384860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.384677    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c572\" (UniqueName: \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.384860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.384759    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.408803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.394566    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.408803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.397222    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.430802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.413451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c572\" (UniqueName: \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\") pod \"maas-api-revoke-sub-45216106-whqm2\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.434800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.431952    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.450151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.449060    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:44.727873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.722862    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.836614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.833889    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:44.836614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:44.834730    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:11:45.451204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:45.438834    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:45.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:45.504509    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:45.848861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:45.726791    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:45.848861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:45.832650    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:11:46.225023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:46.224977    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:46.660033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:46.548904    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:11:46.876805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:46.876344    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:11:47.120311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.120281    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:11:47.120807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.120764    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"]
Nov 01 20:11:47.120922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.120853    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:11:47.141099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.141071    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:47.860864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.860824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2" event={"ID":"96290b8e-b541-473a-9870-2c7a29eb9881","Type":"ContainerStarted","Data":"1dbeaa47bf274c34f6f42de8b57ddd3bb8d53ecc103b597aa94ebdb1c858c7b5"}
Nov 01 20:11:47.861134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.860881    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2" event={"ID":"96290b8e-b541-473a-9870-2c7a29eb9881","Type":"ContainerStarted","Data":"86dc0ee32b800794c8a539dc66ad6aa6c0adb119d4f01d5a27a5528bdb0bc289"}
Nov 01 20:11:47.877634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:47.877579    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2" podStartSLOduration=4.877565778 podStartE2EDuration="4.877565778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:47.87500299 +0000 UTC m=+795.596081678" watchObservedRunningTime="2026-11-01 20:11:47.877565778 +0000 UTC m=+795.598644454"
Nov 01 20:11:48.872989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:48.872956    5041 generic.go:356] "Generic (PLEG): container finished" podID="96290b8e-b541-473a-9870-2c7a29eb9881" containerID="1dbeaa47bf274c34f6f42de8b57ddd3bb8d53ecc103b597aa94ebdb1c858c7b5" exitCode=0
Nov 01 20:11:48.873197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:48.873040    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2" event={"ID":"96290b8e-b541-473a-9870-2c7a29eb9881","Type":"ContainerDied","Data":"1dbeaa47bf274c34f6f42de8b57ddd3bb8d53ecc103b597aa94ebdb1c858c7b5"}
Nov 01 20:11:50.111863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.111835    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:50.133546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.133472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\") pod \"96290b8e-b541-473a-9870-2c7a29eb9881\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") "
Nov 01 20:11:50.133546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.133536    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\") pod \"96290b8e-b541-473a-9870-2c7a29eb9881\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") "
Nov 01 20:11:50.133757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.133572    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\" (UniqueName: \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\") pod \"96290b8e-b541-473a-9870-2c7a29eb9881\" (UID: \"96290b8e-b541-473a-9870-2c7a29eb9881\") "
Nov 01 20:11:50.133958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.133921    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca" pod "96290b8e-b541-473a-9870-2c7a29eb9881" (UID: "96290b8e-b541-473a-9870-2c7a29eb9881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:11:50.134093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.134071    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:50.135715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.135687    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca" pod "96290b8e-b541-473a-9870-2c7a29eb9881" (UID: "96290b8e-b541-473a-9870-2c7a29eb9881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:11:50.135841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.135791    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572" pod "96290b8e-b541-473a-9870-2c7a29eb9881" (UID: "96290b8e-b541-473a-9870-2c7a29eb9881"). InnerVolumeSpecName "kube-api-access-2c572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:11:50.235818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.235742    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96290b8e-b541-473a-9870-2c7a29eb9881-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:50.235818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.235798    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c572\" (UniqueName: \"kubernetes.io/projected/96290b8e-b541-473a-9870-2c7a29eb9881-kube-api-access-2c572\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:11:50.895702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.895618    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"
Nov 01 20:11:50.895702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.895643    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2" event={"ID":"96290b8e-b541-473a-9870-2c7a29eb9881","Type":"ContainerDied","Data":"86dc0ee32b800794c8a539dc66ad6aa6c0adb119d4f01d5a27a5528bdb0bc289"}
Nov 01 20:11:50.895702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:50.895692    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86dc0ee32b800794c8a539dc66ad6aa6c0adb119d4f01d5a27a5528bdb0bc289"
Nov 01 20:11:53.791805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:53.788872    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:53.933924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:53.933875    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.141986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:54.141881    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.239801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:54.238857    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:54.748487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:54.747973    5041 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:11:54.748487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:54.748043    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.032861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.007893    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.032861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.008532    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:11:55.266514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.266456    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:55.273949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.272788    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:11:55.351804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.351757    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:55.352097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:55.352080    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:11:56.023805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:56.016933    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:11:56.403248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:56.402214    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:56.403248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:56.402604    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:11:57.013394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:57.013358    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:11:57.409142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:57.409095    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:11:57.410113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:57.410061    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:11:58.151885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:58.134405    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:11:58.233618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:58.233585    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"]
Nov 01 20:11:58.240191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:58.240165    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:11:58.248970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:58.248940    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af23ea43-s7blm"]
Nov 01 20:11:58.429914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:58.429833    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d996cd75-5930-4285-9803-5a315b1c8097" path="/var/lib/kubelet/pods/d996cd75-5930-4285-9803-5a315b1c8097/volumes"
Nov 01 20:11:58.544803 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:11:58.544754    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:11:59.757770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.757732    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"]
Nov 01 20:11:59.759260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.759229    5041 cpu_manager.go:414] "containerMap: removing container" podUID="96290b8e-b541-473a-9870-2c7a29eb9881" containerName="revoke-keys"
Nov 01 20:11:59.759260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.759259    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96290b8e-b541-473a-9870-2c7a29eb9881" containerName="revoke-keys"
Nov 01 20:11:59.759457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.759423    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="96290b8e-b541-473a-9870-2c7a29eb9881" containerName="revoke-keys"
Nov 01 20:11:59.766052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.766028    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.772445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.771336    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"]
Nov 01 20:11:59.772575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.772515    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:11:59.772653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.772638    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:11:59.772713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.772697    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:11:59.779440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.772773    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:11:59.896266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.896223    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.896469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.896439    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.896624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.896604    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qqq\" (UniqueName: \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.997592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.997554    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.997812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.997676    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qqq\" (UniqueName: \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.997812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.997734    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:11:59.998458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:11:59.998435    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:00.000038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.000013    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:00.006376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.006354    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qqq\" (UniqueName: \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\") pod \"maas-api-revoke-sub-5625be3c-dqnh5\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:00.092542 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.092507    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:00.381963 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:00.381925    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948b9504_de7d_46c9_a928_ad3e9680a183.slice/crio-d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e WatchSource:0}: Error finding container d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e: Status 404 returned error can't find the container with id d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e
Nov 01 20:12:00.383392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.383353    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"]
Nov 01 20:12:00.770948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.770913    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5" event={"ID":"948b9504-de7d-46c9-a928-ad3e9680a183","Type":"ContainerStarted","Data":"03df1ae0237ec4b749e360eb1c3bdbfc8eb261fb6fcd282b514201aef6cba0f8"}
Nov 01 20:12:00.771452 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.771007    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5" event={"ID":"948b9504-de7d-46c9-a928-ad3e9680a183","Type":"ContainerStarted","Data":"d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e"}
Nov 01 20:12:00.786792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:00.786721    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5" podStartSLOduration=1.786707157 podStartE2EDuration="1.786707157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:00.786359477 +0000 UTC m=+808.507438161" watchObservedRunningTime="2026-11-01 20:12:00.786707157 +0000 UTC m=+808.507785833"
Nov 01 20:12:01.781512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:01.781472    5041 generic.go:356] "Generic (PLEG): container finished" podID="948b9504-de7d-46c9-a928-ad3e9680a183" containerID="03df1ae0237ec4b749e360eb1c3bdbfc8eb261fb6fcd282b514201aef6cba0f8" exitCode=0
Nov 01 20:12:01.781917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:01.781557    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5" event={"ID":"948b9504-de7d-46c9-a928-ad3e9680a183","Type":"ContainerDied","Data":"03df1ae0237ec4b749e360eb1c3bdbfc8eb261fb6fcd282b514201aef6cba0f8"}
Nov 01 20:12:02.996543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:02.996507    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:03.164588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.164501    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\") pod \"948b9504-de7d-46c9-a928-ad3e9680a183\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") "
Nov 01 20:12:03.164733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.164594    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\") pod \"948b9504-de7d-46c9-a928-ad3e9680a183\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") "
Nov 01 20:12:03.164733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.164659    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\" (UniqueName: \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\") pod \"948b9504-de7d-46c9-a928-ad3e9680a183\" (UID: \"948b9504-de7d-46c9-a928-ad3e9680a183\") "
Nov 01 20:12:03.164898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.164877    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca" pod "948b9504-de7d-46c9-a928-ad3e9680a183" (UID: "948b9504-de7d-46c9-a928-ad3e9680a183"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:03.165174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.165155    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:03.166610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.166578    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca" pod "948b9504-de7d-46c9-a928-ad3e9680a183" (UID: "948b9504-de7d-46c9-a928-ad3e9680a183"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:03.166873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.166848    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq" pod "948b9504-de7d-46c9-a928-ad3e9680a183" (UID: "948b9504-de7d-46c9-a928-ad3e9680a183"). InnerVolumeSpecName "kube-api-access-87qqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:03.266625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.266590    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b9504-de7d-46c9-a928-ad3e9680a183-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:03.266625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.266624    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qqq\" (UniqueName: \"kubernetes.io/projected/948b9504-de7d-46c9-a928-ad3e9680a183-kube-api-access-87qqq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:03.803284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.803253    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"
Nov 01 20:12:03.803453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.803254    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5" event={"ID":"948b9504-de7d-46c9-a928-ad3e9680a183","Type":"ContainerDied","Data":"d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e"}
Nov 01 20:12:03.803453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:03.803417    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f84b321b7f86ffe7e073a76d304a0524f8001ca967487703180b784566c23e"
Nov 01 20:12:06.265498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:06.264897    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:06.472272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:06.471895    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:12:06.472272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:06.471978    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:06.798095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:06.710974    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:07.269029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.268911    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:07.270469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.270427    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:12:07.417817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.417256    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:12:07.615707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.615675    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:12:07.804037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.803962    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:12:07.833919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:07.833868    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:08.464125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.464089    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"]
Nov 01 20:12:08.466796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.466757    5041 cpu_manager.go:414] "containerMap: removing container" podUID="948b9504-de7d-46c9-a928-ad3e9680a183" containerName="revoke-keys"
Nov 01 20:12:08.466959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.466944    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948b9504-de7d-46c9-a928-ad3e9680a183" containerName="revoke-keys"
Nov 01 20:12:08.467483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.467463    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="948b9504-de7d-46c9-a928-ad3e9680a183" containerName="revoke-keys"
Nov 01 20:12:08.473757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.473735    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"]
Nov 01 20:12:08.473870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.473855    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.477385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.477364    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:12:08.479223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.479202    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:12:08.479338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.479282    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:12:08.479338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.479308    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:12:08.511616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.511581    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncht\" (UniqueName: \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.511935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.511913    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.512198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.512179    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.613512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.613468    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncht\" (UniqueName: \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.613714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.613577    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.613797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.613721    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.614579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.614554    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.616522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.616501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.621100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.621079    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncht\" (UniqueName: \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\") pod \"maas-api-revoke-sub-335aa786-vt495\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:08.794445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:08.794179    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:09.085312 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:09.085274    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7de724_87ff_4748_b6e7_4cde95898864.slice/crio-47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386 WatchSource:0}: Error finding container 47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386: Status 404 returned error can't find the container with id 47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386
Nov 01 20:12:09.089534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:09.089503    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"]
Nov 01 20:12:09.954242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:09.954205    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"]
Nov 01 20:12:09.960302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:09.960272    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:09.963563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:09.963535    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"]
Nov 01 20:12:10.056667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.056630    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.056834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.056812    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.057013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.056995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwl2t\" (UniqueName: \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.076766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.076728    5041 generic.go:356] "Generic (PLEG): container finished" podID="cd7de724-87ff-4748-b6e7-4cde95898864" containerID="64dd1d761ad5e322fb48131d1e58599874b6b4a47001089c5cbc4a6464815259" exitCode=0
Nov 01 20:12:10.076944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.076798    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495" event={"ID":"cd7de724-87ff-4748-b6e7-4cde95898864","Type":"ContainerDied","Data":"64dd1d761ad5e322fb48131d1e58599874b6b4a47001089c5cbc4a6464815259"}
Nov 01 20:12:10.076944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.076861    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495" event={"ID":"cd7de724-87ff-4748-b6e7-4cde95898864","Type":"ContainerStarted","Data":"47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386"}
Nov 01 20:12:10.158756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.158717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.158966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.158820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.158966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.158945    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwl2t\" (UniqueName: \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.159580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.159552    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.161229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.161208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.167465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.167444    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwl2t\" (UniqueName: \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\") pod \"maas-api-revoke-sub-f0f431ba-swdrt\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.288699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.288667    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:10.579578 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:10.579538    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9652689f_1204_450e_a9cb_a5901386796f.slice/crio-e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15 WatchSource:0}: Error finding container e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15: Status 404 returned error can't find the container with id e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15
Nov 01 20:12:10.582064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:10.582039    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"]
Nov 01 20:12:11.087979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.087941    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt" event={"ID":"9652689f-1204-450e-a9cb-a5901386796f","Type":"ContainerStarted","Data":"76df7018dfe1c1374a3d43d8b726e8cda01f81aba37318e941a9b42bb970cbac"}
Nov 01 20:12:11.088414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.088007    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt" event={"ID":"9652689f-1204-450e-a9cb-a5901386796f","Type":"ContainerStarted","Data":"e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15"}
Nov 01 20:12:11.107417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.107323    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt" podStartSLOduration=2.107308537 podStartE2EDuration="2.107308537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:11.103677187 +0000 UTC m=+818.824755865" watchObservedRunningTime="2026-11-01 20:12:11.107308537 +0000 UTC m=+818.828387213"
Nov 01 20:12:11.259276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.259252    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:11.381511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.381435    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\") pod \"cd7de724-87ff-4748-b6e7-4cde95898864\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") "
Nov 01 20:12:11.381511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.381481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\") pod \"cd7de724-87ff-4748-b6e7-4cde95898864\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") "
Nov 01 20:12:11.381832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.381811    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\" (UniqueName: \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\") pod \"cd7de724-87ff-4748-b6e7-4cde95898864\" (UID: \"cd7de724-87ff-4748-b6e7-4cde95898864\") "
Nov 01 20:12:11.381832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.381820    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca" pod "cd7de724-87ff-4748-b6e7-4cde95898864" (UID: "cd7de724-87ff-4748-b6e7-4cde95898864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:11.382305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.382285    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:11.442191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.442157    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca" pod "cd7de724-87ff-4748-b6e7-4cde95898864" (UID: "cd7de724-87ff-4748-b6e7-4cde95898864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:11.442332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.442188    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht" pod "cd7de724-87ff-4748-b6e7-4cde95898864" (UID: "cd7de724-87ff-4748-b6e7-4cde95898864"). InnerVolumeSpecName "kube-api-access-nncht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:11.483224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.483192    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nncht\" (UniqueName: \"kubernetes.io/projected/cd7de724-87ff-4748-b6e7-4cde95898864-kube-api-access-nncht\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:11.483224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:11.483219    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd7de724-87ff-4748-b6e7-4cde95898864-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:12.098477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:12.098442    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"
Nov 01 20:12:12.098477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:12.098460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495" event={"ID":"cd7de724-87ff-4748-b6e7-4cde95898864","Type":"ContainerDied","Data":"47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386"}
Nov 01 20:12:12.099058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:12.098508    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47331cd00947570689983f02eb4c487730fd662db363631768610babc2e7d386"
Nov 01 20:12:12.100084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:12.100055    5041 generic.go:356] "Generic (PLEG): container finished" podID="9652689f-1204-450e-a9cb-a5901386796f" containerID="76df7018dfe1c1374a3d43d8b726e8cda01f81aba37318e941a9b42bb970cbac" exitCode=0
Nov 01 20:12:12.100214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:12.100127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt" event={"ID":"9652689f-1204-450e-a9cb-a5901386796f","Type":"ContainerDied","Data":"76df7018dfe1c1374a3d43d8b726e8cda01f81aba37318e941a9b42bb970cbac"}
Nov 01 20:12:13.297544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.297513    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:13.407093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.407000    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\") pod \"9652689f-1204-450e-a9cb-a5901386796f\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") "
Nov 01 20:12:13.407093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.407058    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\") pod \"9652689f-1204-450e-a9cb-a5901386796f\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") "
Nov 01 20:12:13.407333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.407095    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\" (UniqueName: \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\") pod \"9652689f-1204-450e-a9cb-a5901386796f\" (UID: \"9652689f-1204-450e-a9cb-a5901386796f\") "
Nov 01 20:12:13.407435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.407412    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca" pod "9652689f-1204-450e-a9cb-a5901386796f" (UID: "9652689f-1204-450e-a9cb-a5901386796f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:13.407661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.407637    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9652689f-1204-450e-a9cb-a5901386796f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:13.409744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.409721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca" pod "9652689f-1204-450e-a9cb-a5901386796f" (UID: "9652689f-1204-450e-a9cb-a5901386796f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:13.409884 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.409863    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t" pod "9652689f-1204-450e-a9cb-a5901386796f" (UID: "9652689f-1204-450e-a9cb-a5901386796f"). InnerVolumeSpecName "kube-api-access-rwl2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:13.508836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.508796    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9652689f-1204-450e-a9cb-a5901386796f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:13.508836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:13.508835    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwl2t\" (UniqueName: \"kubernetes.io/projected/9652689f-1204-450e-a9cb-a5901386796f-kube-api-access-rwl2t\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:14.119586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:14.119554    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"
Nov 01 20:12:14.119798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:14.119592    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt" event={"ID":"9652689f-1204-450e-a9cb-a5901386796f","Type":"ContainerDied","Data":"e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15"}
Nov 01 20:12:14.119798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:14.119643    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ec33d92ed73a28f5f7ae4865ef4e72065968ca0e4bccd64b1ffc1cb88e1f15"
Nov 01 20:12:20.429448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.429413    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"]
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.429978    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cd7de724-87ff-4748-b6e7-4cde95898864" containerName="revoke-keys"
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.429991    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7de724-87ff-4748-b6e7-4cde95898864" containerName="revoke-keys"
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.430035    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9652689f-1204-450e-a9cb-a5901386796f" containerName="revoke-keys"
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.430040    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9652689f-1204-450e-a9cb-a5901386796f" containerName="revoke-keys"
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.430136    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7de724-87ff-4748-b6e7-4cde95898864" containerName="revoke-keys"
Nov 01 20:12:20.430159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.430153    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9652689f-1204-450e-a9cb-a5901386796f" containerName="revoke-keys"
Nov 01 20:12:20.433421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.433398    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.447388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.447344    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:12:20.448419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.447727    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:12:20.448419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.447831    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:12:20.448419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.447956    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:12:20.450257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.450231    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"]
Nov 01 20:12:20.494063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.494026    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.494424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.494402    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.494821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.494730    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf94\" (UniqueName: \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.604646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.604602    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf94\" (UniqueName: \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.604892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.604674    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.604892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.604816    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.605602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.605574    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.607223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.607204    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.613794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.613745    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf94\" (UniqueName: \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\") pod \"maas-api-revoke-sub-48743828-hlk7x\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:20.760325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:20.760240    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:21.208696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:21.208125    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"]
Nov 01 20:12:22.287803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:22.286823    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x" event={"ID":"b3ffe76f-fc73-4874-9f90-51c0eb0364f3","Type":"ContainerStarted","Data":"0044fdc3b1aee578105c75634c09e4d0e92e41abd0c78572f15e39c32398f090"}
Nov 01 20:12:22.287803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:22.286907    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x" event={"ID":"b3ffe76f-fc73-4874-9f90-51c0eb0364f3","Type":"ContainerStarted","Data":"80a3329af753dc6817e65bdd30fe07509a641902227d08298d3fcbb4b3ac6b49"}
Nov 01 20:12:22.328802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:22.318359    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x" podStartSLOduration=2.318336922 podStartE2EDuration="2.318336922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:22.315251475 +0000 UTC m=+830.036330164" watchObservedRunningTime="2026-11-01 20:12:22.318336922 +0000 UTC m=+830.039415598"
Nov 01 20:12:23.299036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.299002    5041 generic.go:356] "Generic (PLEG): container finished" podID="b3ffe76f-fc73-4874-9f90-51c0eb0364f3" containerID="0044fdc3b1aee578105c75634c09e4d0e92e41abd0c78572f15e39c32398f090" exitCode=0
Nov 01 20:12:23.299455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.299094    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x" event={"ID":"b3ffe76f-fc73-4874-9f90-51c0eb0364f3","Type":"ContainerDied","Data":"0044fdc3b1aee578105c75634c09e4d0e92e41abd0c78572f15e39c32398f090"}
Nov 01 20:12:23.856437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.856403    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:12:23.862216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.862189    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.865768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.865730    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-638f19-openshift-default-dockercfg-k2znv\""
Nov 01 20:12:23.870394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.870368    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:12:23.920463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.919952    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920168    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920208    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920347    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920386    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920495    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920532    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920559    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwklg\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:23.920912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:23.920606    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022036    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022183    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022210    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022306    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022364    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwklg\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022430    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022718    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.022874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022822    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.023101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.022927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.023557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.023530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.023672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.023630    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.024870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.024840    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.024870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.024850    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.049125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.049091    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwklg\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.049272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.049253    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\") pod \"e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") " pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.104540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.104503    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"]
Nov 01 20:12:24.110401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.110343    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.132462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.132425    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"]
Nov 01 20:12:24.191254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.191216    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:24.225872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.225837    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.226065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.225897    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvqh\" (UniqueName: \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.226180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.226148    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.328657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.328011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.328657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.328209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.328657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.328262    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvqh\" (UniqueName: \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.329573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.329515    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.331882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.331857    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.340017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.339870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvqh\" (UniqueName: \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\") pod \"maas-api-revoke-sub-f811fc1d-8qjbz\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.426183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.426140    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:24.543596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.543529    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:24.543930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.543900    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:24.544177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.544102    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:24.557923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.557664    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:12:24.759093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.754845    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:24.881801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.860459    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"]
Nov 01 20:12:24.950046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.950015    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\") pod \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") "
Nov 01 20:12:24.950206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.950106    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\") pod \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") "
Nov 01 20:12:24.950275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.950246    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\" (UniqueName: \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\") pod \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\" (UID: \"b3ffe76f-fc73-4874-9f90-51c0eb0364f3\") "
Nov 01 20:12:24.950498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.950465    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca" pod "b3ffe76f-fc73-4874-9f90-51c0eb0364f3" (UID: "b3ffe76f-fc73-4874-9f90-51c0eb0364f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:24.950723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.950701    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:24.952685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.952658    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94" pod "b3ffe76f-fc73-4874-9f90-51c0eb0364f3" (UID: "b3ffe76f-fc73-4874-9f90-51c0eb0364f3"). InnerVolumeSpecName "kube-api-access-6vf94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:24.953255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:24.953231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca" pod "b3ffe76f-fc73-4874-9f90-51c0eb0364f3" (UID: "b3ffe76f-fc73-4874-9f90-51c0eb0364f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:25.055035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.054983    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:25.055035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.055014    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf94\" (UniqueName: \"kubernetes.io/projected/b3ffe76f-fc73-4874-9f90-51c0eb0364f3-kube-api-access-6vf94\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:25.325890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.325831    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" event={"ID":"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f","Type":"ContainerStarted","Data":"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"}
Nov 01 20:12:25.326087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.325899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" event={"ID":"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f","Type":"ContainerStarted","Data":"dbe8aae9cb0331283bcbfbf7feec8c76dfd9c85d200749e5df481a951b7019a1"}
Nov 01 20:12:25.333801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.328226    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"
Nov 01 20:12:25.333801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.328229    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x" event={"ID":"b3ffe76f-fc73-4874-9f90-51c0eb0364f3","Type":"ContainerDied","Data":"80a3329af753dc6817e65bdd30fe07509a641902227d08298d3fcbb4b3ac6b49"}
Nov 01 20:12:25.333801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.328399    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80a3329af753dc6817e65bdd30fe07509a641902227d08298d3fcbb4b3ac6b49"
Nov 01 20:12:25.333801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.330260    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz" event={"ID":"78a25c60-f530-4989-8274-1ec0bb9d6ea5","Type":"ContainerStarted","Data":"718c196a8bcce3812e24962d31b87053469295acb35315136e596845b49ce005"}
Nov 01 20:12:25.333801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.330304    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz" event={"ID":"78a25c60-f530-4989-8274-1ec0bb9d6ea5","Type":"ContainerStarted","Data":"26f7343080eaa56a3aabbdd17a85a60b496485b931913ddcd479ced11fd2ef22"}
Nov 01 20:12:25.358582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.358506    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podStartSLOduration=2.358483444 podStartE2EDuration="2.358483444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:25.345889441 +0000 UTC m=+833.066968118" watchObservedRunningTime="2026-11-01 20:12:25.358483444 +0000 UTC m=+833.079562122"
Nov 01 20:12:25.376839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:25.376309    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz" podStartSLOduration=1.37629095 podStartE2EDuration="1.37629095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:25.361824369 +0000 UTC m=+833.082903046" watchObservedRunningTime="2026-11-01 20:12:25.37629095 +0000 UTC m=+833.097369626"
Nov 01 20:12:26.192481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:26.192391    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:26.194300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:26.194272    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:26.194427 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:26.194325    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:26.339975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:26.339938    5041 generic.go:356] "Generic (PLEG): container finished" podID="78a25c60-f530-4989-8274-1ec0bb9d6ea5" containerID="718c196a8bcce3812e24962d31b87053469295acb35315136e596845b49ce005" exitCode=0
Nov 01 20:12:26.340380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:26.340012    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz" event={"ID":"78a25c60-f530-4989-8274-1ec0bb9d6ea5","Type":"ContainerDied","Data":"718c196a8bcce3812e24962d31b87053469295acb35315136e596845b49ce005"}
Nov 01 20:12:27.191944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.191909    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:27.192144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.191972    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:27.609028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.609004    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:27.702206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.702167    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\") pod \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") "
Nov 01 20:12:27.702396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.702218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\" (UniqueName: \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\") pod \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") "
Nov 01 20:12:27.702396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.702326    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\") pod \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\" (UID: \"78a25c60-f530-4989-8274-1ec0bb9d6ea5\") "
Nov 01 20:12:27.703009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.702955    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca" pod "78a25c60-f530-4989-8274-1ec0bb9d6ea5" (UID: "78a25c60-f530-4989-8274-1ec0bb9d6ea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:27.704991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.704967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca" pod "78a25c60-f530-4989-8274-1ec0bb9d6ea5" (UID: "78a25c60-f530-4989-8274-1ec0bb9d6ea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:27.705108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.705005    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh" pod "78a25c60-f530-4989-8274-1ec0bb9d6ea5" (UID: "78a25c60-f530-4989-8274-1ec0bb9d6ea5"). InnerVolumeSpecName "kube-api-access-pxvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:27.804590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.804550    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:27.804590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.804579    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvqh\" (UniqueName: \"kubernetes.io/projected/78a25c60-f530-4989-8274-1ec0bb9d6ea5-kube-api-access-pxvqh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:27.804590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:27.804589    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a25c60-f530-4989-8274-1ec0bb9d6ea5-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:28.191620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:28.191545    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:28.191620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:28.191594    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:28.380695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:28.380649    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz" event={"ID":"78a25c60-f530-4989-8274-1ec0bb9d6ea5","Type":"ContainerDied","Data":"26f7343080eaa56a3aabbdd17a85a60b496485b931913ddcd479ced11fd2ef22"}
Nov 01 20:12:28.380919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:28.380713    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f7343080eaa56a3aabbdd17a85a60b496485b931913ddcd479ced11fd2ef22"
Nov 01 20:12:28.380919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:28.380671    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"
Nov 01 20:12:28.545825 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:12:28.541842    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:12:29.191952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:29.191921    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:29.192338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:29.191970    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:30.192117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:30.192077    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:30.192666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:30.192161    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:31.192423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:31.192384    5041 patch_prober.go:29] interesting pod/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Nov 01 20:12:31.192867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:31.192447    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Nov 01 20:12:32.201627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:32.201593    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:32.203949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:32.203925    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:32.223286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:32.219074    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.616834    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"]
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.617773    5041 cpu_manager.go:414] "containerMap: removing container" podUID="78a25c60-f530-4989-8274-1ec0bb9d6ea5" containerName="revoke-keys"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.617810    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a25c60-f530-4989-8274-1ec0bb9d6ea5" containerName="revoke-keys"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.617831    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b3ffe76f-fc73-4874-9f90-51c0eb0364f3" containerName="revoke-keys"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.617840    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ffe76f-fc73-4874-9f90-51c0eb0364f3" containerName="revoke-keys"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.618022    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ffe76f-fc73-4874-9f90-51c0eb0364f3" containerName="revoke-keys"
Nov 01 20:12:34.620998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.618048    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a25c60-f530-4989-8274-1ec0bb9d6ea5" containerName="revoke-keys"
Nov 01 20:12:34.627555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.627529    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.637447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.637420    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"]
Nov 01 20:12:34.642484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.639038    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:12:34.644892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.642885    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:12:34.644892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.642970    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:12:34.644892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.643060    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:12:34.740398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.737485    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.740398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.737592    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9vsb\" (UniqueName: \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.740398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.737699    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.844191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.844164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.844482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.844459    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.844581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.844544    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9vsb\" (UniqueName: \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.845468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.845441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.855882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.855850    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9vsb\" (UniqueName: \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:34.869649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:34.869616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9309b76-cbrtc\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:35.080644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.080608    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:35.319426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.319393    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"]
Nov 01 20:12:35.325412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.325387    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.332935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.332869    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"]
Nov 01 20:12:35.368897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.368871    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"]
Nov 01 20:12:35.453585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.453549    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.453748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.453671    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.453818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.453769    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-864qn\" (UniqueName: \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.555551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.555521    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.555708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.555573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.555708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.555618    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-864qn\" (UniqueName: \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.556523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.556496    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.558857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.558456    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.563609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.563588    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-864qn\" (UniqueName: \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\") pod \"maas-api-revoke-sub-540f116a-hzw6c\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.650926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.650898    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:35.775274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.775228    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc" event={"ID":"317db877-c521-49d5-bf59-c75e32634333","Type":"ContainerStarted","Data":"84f14528d067b4df69d8ad4d9ae671d0ae488f2ac736f8cbbb311d804e7202fb"}
Nov 01 20:12:35.775438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.775300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc" event={"ID":"317db877-c521-49d5-bf59-c75e32634333","Type":"ContainerStarted","Data":"342b1647313e0c2fca5dc23f3c1546323eedd24986a520525d87990a3125ae11"}
Nov 01 20:12:35.797159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.797098    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc" podStartSLOduration=1.797079268 podStartE2EDuration="1.797079268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:35.79107909 +0000 UTC m=+843.512157763" watchObservedRunningTime="2026-11-01 20:12:35.797079268 +0000 UTC m=+843.518157947"
Nov 01 20:12:35.961230 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:35.961199    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04c4ff0_a311_4482_9cc2_332b0e8b409d.slice/crio-f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee WatchSource:0}: Error finding container f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee: Status 404 returned error can't find the container with id f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee
Nov 01 20:12:35.961597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:35.961573    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"]
Nov 01 20:12:36.785492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:36.785448    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c" event={"ID":"e04c4ff0-a311-4482-9cc2-332b0e8b409d","Type":"ContainerStarted","Data":"f4754e0f041ad25744796cdcfd192027caaa2b70437eade5bc5d1f7857557957"}
Nov 01 20:12:36.785990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:36.785551    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c" event={"ID":"e04c4ff0-a311-4482-9cc2-332b0e8b409d","Type":"ContainerStarted","Data":"f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee"}
Nov 01 20:12:36.787560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:36.787532    5041 generic.go:356] "Generic (PLEG): container finished" podID="317db877-c521-49d5-bf59-c75e32634333" containerID="84f14528d067b4df69d8ad4d9ae671d0ae488f2ac736f8cbbb311d804e7202fb" exitCode=0
Nov 01 20:12:36.787654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:36.787618    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc" event={"ID":"317db877-c521-49d5-bf59-c75e32634333","Type":"ContainerDied","Data":"84f14528d067b4df69d8ad4d9ae671d0ae488f2ac736f8cbbb311d804e7202fb"}
Nov 01 20:12:36.805563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:36.805512    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c" podStartSLOduration=1.80549856 podStartE2EDuration="1.80549856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:36.801298275 +0000 UTC m=+844.522376962" watchObservedRunningTime="2026-11-01 20:12:36.80549856 +0000 UTC m=+844.526577236"
Nov 01 20:12:37.797836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:37.797772    5041 generic.go:356] "Generic (PLEG): container finished" podID="e04c4ff0-a311-4482-9cc2-332b0e8b409d" containerID="f4754e0f041ad25744796cdcfd192027caaa2b70437eade5bc5d1f7857557957" exitCode=0
Nov 01 20:12:37.798275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:37.797853    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c" event={"ID":"e04c4ff0-a311-4482-9cc2-332b0e8b409d","Type":"ContainerDied","Data":"f4754e0f041ad25744796cdcfd192027caaa2b70437eade5bc5d1f7857557957"}
Nov 01 20:12:37.972380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:37.972348    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:38.090068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.089987    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\") pod \"317db877-c521-49d5-bf59-c75e32634333\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") "
Nov 01 20:12:38.090233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.090087    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\" (UniqueName: \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\") pod \"317db877-c521-49d5-bf59-c75e32634333\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") "
Nov 01 20:12:38.090233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.090201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\") pod \"317db877-c521-49d5-bf59-c75e32634333\" (UID: \"317db877-c521-49d5-bf59-c75e32634333\") "
Nov 01 20:12:38.090576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.090552    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca" pod "317db877-c521-49d5-bf59-c75e32634333" (UID: "317db877-c521-49d5-bf59-c75e32634333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:38.092120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.092098    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb" pod "317db877-c521-49d5-bf59-c75e32634333" (UID: "317db877-c521-49d5-bf59-c75e32634333"). InnerVolumeSpecName "kube-api-access-h9vsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:38.092186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.092116    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca" pod "317db877-c521-49d5-bf59-c75e32634333" (UID: "317db877-c521-49d5-bf59-c75e32634333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:38.191553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.191518    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317db877-c521-49d5-bf59-c75e32634333-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:38.191553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.191547    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9vsb\" (UniqueName: \"kubernetes.io/projected/317db877-c521-49d5-bf59-c75e32634333-kube-api-access-h9vsb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:38.191553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.191557    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317db877-c521-49d5-bf59-c75e32634333-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:38.807538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.807507    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"
Nov 01 20:12:38.807946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.807539    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc" event={"ID":"317db877-c521-49d5-bf59-c75e32634333","Type":"ContainerDied","Data":"342b1647313e0c2fca5dc23f3c1546323eedd24986a520525d87990a3125ae11"}
Nov 01 20:12:38.807946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:38.807585    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342b1647313e0c2fca5dc23f3c1546323eedd24986a520525d87990a3125ae11"
Nov 01 20:12:39.003985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.003957    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:39.103069 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.102992    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\" (UniqueName: \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\") pod \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") "
Nov 01 20:12:39.103209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.103164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\") pod \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") "
Nov 01 20:12:39.103254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.103219    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\") pod \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\" (UID: \"e04c4ff0-a311-4482-9cc2-332b0e8b409d\") "
Nov 01 20:12:39.103515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.103487    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca" pod "e04c4ff0-a311-4482-9cc2-332b0e8b409d" (UID: "e04c4ff0-a311-4482-9cc2-332b0e8b409d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:39.103611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.103584    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:39.105112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.105084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn" pod "e04c4ff0-a311-4482-9cc2-332b0e8b409d" (UID: "e04c4ff0-a311-4482-9cc2-332b0e8b409d"). InnerVolumeSpecName "kube-api-access-864qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:39.105335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.105316    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca" pod "e04c4ff0-a311-4482-9cc2-332b0e8b409d" (UID: "e04c4ff0-a311-4482-9cc2-332b0e8b409d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:39.204709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.204679    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e04c4ff0-a311-4482-9cc2-332b0e8b409d-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:39.204709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.204708    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-864qn\" (UniqueName: \"kubernetes.io/projected/e04c4ff0-a311-4482-9cc2-332b0e8b409d-kube-api-access-864qn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:39.818112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.818087    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"
Nov 01 20:12:39.818112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.818112    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c" event={"ID":"e04c4ff0-a311-4482-9cc2-332b0e8b409d","Type":"ContainerDied","Data":"f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee"}
Nov 01 20:12:39.818541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:39.818158    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8cd9d2c827df773df19558d6ffd28c19d7b1f1799c53555f796adec406635ee"
Nov 01 20:12:43.427407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.427326    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:12:43.428180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428160    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e04c4ff0-a311-4482-9cc2-332b0e8b409d" containerName="revoke-keys"
Nov 01 20:12:43.428180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428181    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04c4ff0-a311-4482-9cc2-332b0e8b409d" containerName="revoke-keys"
Nov 01 20:12:43.428347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428217    5041 cpu_manager.go:414] "containerMap: removing container" podUID="317db877-c521-49d5-bf59-c75e32634333" containerName="revoke-keys"
Nov 01 20:12:43.428347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428225    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317db877-c521-49d5-bf59-c75e32634333" containerName="revoke-keys"
Nov 01 20:12:43.428447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428373    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="317db877-c521-49d5-bf59-c75e32634333" containerName="revoke-keys"
Nov 01 20:12:43.428447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.428400    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04c4ff0-a311-4482-9cc2-332b0e8b409d" containerName="revoke-keys"
Nov 01 20:12:43.432456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.432436    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.438195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.438163    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5370bb-openshift-default-dockercfg-m26p7\""
Nov 01 20:12:43.442130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.442101    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:12:43.557894 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.557840    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.557972    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558013    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558073    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558171    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558218    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558271    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558379    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.558554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.558536    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgtg\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660292    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660366    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660414    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660443    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660584    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660731    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660746    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgtg\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.660923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.660852    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.661046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.661023    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.661120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.661097    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.661462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.661441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.662772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.662735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.663147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.663129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.667634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.667611    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.668275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.668254    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgtg\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:43.755729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:43.755648    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:44.047930 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:44.047898    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43a2db1_474d_41b3_af31_61702130514b.slice/crio-f18c915e7db7b0d72624a89bdec3cc58b89bcbf983c00804a0808ab15d7f5949 WatchSource:0}: Error finding container f18c915e7db7b0d72624a89bdec3cc58b89bcbf983c00804a0808ab15d7f5949: Status 404 returned error can't find the container with id f18c915e7db7b0d72624a89bdec3cc58b89bcbf983c00804a0808ab15d7f5949
Nov 01 20:12:44.050654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.050617    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:44.050765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.050690    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:44.050765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.050720    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:12:44.051037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.051011    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:12:44.884165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.884120    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" event={"ID":"c43a2db1-474d-41b3-af31-61702130514b","Type":"ContainerStarted","Data":"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"}
Nov 01 20:12:44.884636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.884191    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" event={"ID":"c43a2db1-474d-41b3-af31-61702130514b","Type":"ContainerStarted","Data":"f18c915e7db7b0d72624a89bdec3cc58b89bcbf983c00804a0808ab15d7f5949"}
Nov 01 20:12:44.909742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:44.909689    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podStartSLOduration=1.909674469 podStartE2EDuration="1.909674469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:44.904721524 +0000 UTC m=+852.625800205" watchObservedRunningTime="2026-11-01 20:12:44.909674469 +0000 UTC m=+852.630753146"
Nov 01 20:12:45.761802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:45.761720    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:45.762023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:45.761867    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:45.762023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:45.761914    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:46.028891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:46.028798    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"]
Nov 01 20:12:46.041114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:46.041085    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a66eec-ghdtq"]
Nov 01 20:12:46.435800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:46.434347    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09278f6-f095-4a02-a3ab-08f3f4a21bcf" path="/var/lib/kubelet/pods/e09278f6-f095-4a02-a3ab-08f3f4a21bcf/volumes"
Nov 01 20:12:46.758902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:46.756327    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:46.758902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:46.756407    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:47.703804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:47.701460    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:12:47.772927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:47.761149    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:47.772927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:47.761207    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:48.468805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.461892    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.642470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.642418    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 01 20:12:48.707795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.707725    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:48.733261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.733164    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:12:48.786803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.761913    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:48.786803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:48.761999    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:49.338652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.338606    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 01 20:12:49.468333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.468288    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:49.469311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.469286    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:12:49.667861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.642657    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:12:49.697267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.671319    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:12:49.762933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.762897    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:49.839415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.778832    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:12:49.840294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:49.840245    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:50.026549 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.026462    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:12:50.222186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.217742    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:12:50.235306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.235273    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:12:50.550022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.549987    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"]
Nov 01 20:12:50.583033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.582997    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"]
Nov 01 20:12:50.583347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.583329    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.590128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.589920    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:12:50.590128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.590055    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:12:50.590522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.590503    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:12:50.592725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.591867    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:12:50.633317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.633289    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.633486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.633359    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjxp\" (UniqueName: \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.633581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.633554    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.734739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.734699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjxp\" (UniqueName: \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.734936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.734814    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.735028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.734983    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.735533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.735509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.737326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.737306    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.747447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.747419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjxp\" (UniqueName: \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\") pod \"maas-api-revoke-sub-83044d3b-m9jpr\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:50.760256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.760227    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:50.760493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.760469    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:50.910134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:50.910058    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:51.323961 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:12:51.323917    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc87e8b_9c71_45a8_9daf_2d25ccaf3985.slice/crio-15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff WatchSource:0}: Error finding container 15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff: Status 404 returned error can't find the container with id 15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff
Nov 01 20:12:51.325453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:51.325422    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"]
Nov 01 20:12:51.405137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:51.398316    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr" event={"ID":"afc87e8b-9c71-45a8-9daf-2d25ccaf3985","Type":"ContainerStarted","Data":"15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff"}
Nov 01 20:12:51.758305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:51.758209    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Nov 01 20:12:51.758659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:51.758626    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Nov 01 20:12:52.413157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:52.413109    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr" event={"ID":"afc87e8b-9c71-45a8-9daf-2d25ccaf3985","Type":"ContainerStarted","Data":"4d377f1d92d8631f2b6df45ce05cced024e901b46efa5df8e2ece90243ad7f15"}
Nov 01 20:12:52.431440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:52.431375    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr" podStartSLOduration=2.4313566189999998 podStartE2EDuration="2.431356619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:52.429584324 +0000 UTC m=+860.150663024" watchObservedRunningTime="2026-11-01 20:12:52.431356619 +0000 UTC m=+860.152435296"
Nov 01 20:12:53.428415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:53.428375    5041 generic.go:356] "Generic (PLEG): container finished" podID="afc87e8b-9c71-45a8-9daf-2d25ccaf3985" containerID="4d377f1d92d8631f2b6df45ce05cced024e901b46efa5df8e2ece90243ad7f15" exitCode=0
Nov 01 20:12:53.429213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:53.428621    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr" event={"ID":"afc87e8b-9c71-45a8-9daf-2d25ccaf3985","Type":"ContainerDied","Data":"4d377f1d92d8631f2b6df45ce05cced024e901b46efa5df8e2ece90243ad7f15"}
Nov 01 20:12:53.431660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:53.431446    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:53.433130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:53.433097    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:53.434002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:53.433982    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:12:54.069657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.058832    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"]
Nov 01 20:12:54.069657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.069117    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b54cc-rm4z8"]
Nov 01 20:12:54.434865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.433916    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd72ffa-1ecb-446e-9c62-17f3158ee0a5" path="/var/lib/kubelet/pods/fbd72ffa-1ecb-446e-9c62-17f3158ee0a5/volumes"
Nov 01 20:12:54.697487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.697423    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:54.813801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.813199    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca" pod "afc87e8b-9c71-45a8-9daf-2d25ccaf3985" (UID: "afc87e8b-9c71-45a8-9daf-2d25ccaf3985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:12:54.846844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.799184    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\") pod \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") "
Nov 01 20:12:54.847137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.847119    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\" (UniqueName: \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\") pod \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") "
Nov 01 20:12:54.866840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.847490    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\") pod \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\" (UID: \"afc87e8b-9c71-45a8-9daf-2d25ccaf3985\") "
Nov 01 20:12:54.897535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.897491    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp" pod "afc87e8b-9c71-45a8-9daf-2d25ccaf3985" (UID: "afc87e8b-9c71-45a8-9daf-2d25ccaf3985"). InnerVolumeSpecName "kube-api-access-9hjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:12:54.907588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.906044    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:54.907588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.906082    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjxp\" (UniqueName: \"kubernetes.io/projected/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-kube-api-access-9hjxp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:54.907588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:54.906954    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca" pod "afc87e8b-9c71-45a8-9daf-2d25ccaf3985" (UID: "afc87e8b-9c71-45a8-9daf-2d25ccaf3985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:12:55.009341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:55.009249    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afc87e8b-9c71-45a8-9daf-2d25ccaf3985-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:12:55.462287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:55.462257    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"
Nov 01 20:12:55.462697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:55.462290    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr" event={"ID":"afc87e8b-9c71-45a8-9daf-2d25ccaf3985","Type":"ContainerDied","Data":"15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff"}
Nov 01 20:12:55.462697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:55.462350    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15535c70c2018302c771c3b4c2a39b67fb8aa8f950e61f23aaf78fd7f0be76ff"
Nov 01 20:12:58.545037 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:12:58.544992    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:12:59.026526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:59.026446    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"]
Nov 01 20:12:59.031709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:12:59.031685    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab37f75-lknbk"]
Nov 01 20:13:00.429429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:00.429395    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59401097-6dc5-4ef1-9b01-f29a3dc81f9f" path="/var/lib/kubelet/pods/59401097-6dc5-4ef1-9b01-f29a3dc81f9f/volumes"
Nov 01 20:13:05.036824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:05.034875    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:13:09.054496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.054457    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"]
Nov 01 20:13:09.063199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.063171    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa76a8d-rjvnh"]
Nov 01 20:13:09.265125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.265092    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:13:09.265871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.265843    5041 cpu_manager.go:414] "containerMap: removing container" podUID="afc87e8b-9c71-45a8-9daf-2d25ccaf3985" containerName="revoke-keys"
Nov 01 20:13:09.265871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.265867    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc87e8b-9c71-45a8-9daf-2d25ccaf3985" containerName="revoke-keys"
Nov 01 20:13:09.266093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.266016    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc87e8b-9c71-45a8-9daf-2d25ccaf3985" containerName="revoke-keys"
Nov 01 20:13:09.271572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.271551    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.279595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.279567    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bccce1-openshift-default-dockercfg-5zk62\""
Nov 01 20:13:09.282180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.282161    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:13:09.302942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.302914    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.302973    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303064    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303092    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303154    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303206    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48thb\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303247    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303348    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.303397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.303378    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.415250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.406691    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.416165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.416135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.416862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.416842    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.417033    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426169    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48thb\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426243    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426306    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426354    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426483    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.426004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.427142    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.427332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.416760    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.429104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.425340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.459800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.457738    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.459800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.458037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.459800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.458570    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.475720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.474288    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48thb\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\") pod \"e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") " pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:09.589685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.589644    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:10.002944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:09.999762    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:13:10.008011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.005912    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:10.008011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.006002    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:10.008011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.006046    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:10.429684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.429654    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf" path="/var/lib/kubelet/pods/3ac1d6c0-ccd7-4191-abc4-bd753c46e9bf/volumes"
Nov 01 20:13:10.771477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.771372    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerStarted","Data":"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"}
Nov 01 20:13:10.771650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.771483    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerStarted","Data":"923cc60bd69b8d70d643eae641b68a436bc6e1342d6a42011b6e52bf3d64d08d"}
Nov 01 20:13:10.795447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:10.795398    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podStartSLOduration=1.795380272 podStartE2EDuration="1.795380272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:10.791109822 +0000 UTC m=+878.512188489" watchObservedRunningTime="2026-11-01 20:13:10.795380272 +0000 UTC m=+878.516458948"
Nov 01 20:13:11.101798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.100834    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"]
Nov 01 20:13:11.125821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.125168    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b2faa7-9qf64"]
Nov 01 20:13:11.589843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.589809    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:11.595295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.595269    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:11.779857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.779824    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:11.780705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:11.780688    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:12.429112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:12.429079    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371" path="/var/lib/kubelet/pods/7c7ff6c6-93c3-4b0a-8cbb-9cfb252e6371/volumes"
Nov 01 20:13:21.035758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:21.035721    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"]
Nov 01 20:13:21.041134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:21.041107    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52ed9af5-s5dks"]
Nov 01 20:13:22.429488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:22.429452    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb7410e-b652-4eb6-a590-a466be89d726" path="/var/lib/kubelet/pods/ccb7410e-b652-4eb6-a590-a466be89d726/volumes"
Nov 01 20:13:26.023210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:26.023168    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"]
Nov 01 20:13:26.028952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:26.028926    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4ecaa5-mkq68"]
Nov 01 20:13:26.429117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:26.429080    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6fb124-d3be-4b60-89a6-e3134f94ad59" path="/var/lib/kubelet/pods/6b6fb124-d3be-4b60-89a6-e3134f94ad59/volumes"
Nov 01 20:13:28.542094 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:13:28.542064    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:13:31.024092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:31.024056    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"]
Nov 01 20:13:31.030461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:31.030432    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee0efc2-kncss"]
Nov 01 20:13:32.442409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:32.442369    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77" path="/var/lib/kubelet/pods/0fb77bed-6b07-4d7c-ad18-b8f0e5dabf77/volumes"
Nov 01 20:13:32.639546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:32.639510    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:13:32.642139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:32.642117    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:13:33.023238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:33.023202    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"]
Nov 01 20:13:33.029504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:33.029471    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e12204d-hfvmf"]
Nov 01 20:13:33.327981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:33.327950    5041 scope.go:122] "RemoveContainer" containerID="ef670f2c39a98df7f3a53668e36a624fb61695a287a0b8df6a8efad5441ccfa9"
Nov 01 20:13:34.429762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:34.429729    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d28eec-663d-4019-b9d2-b8ea9dec9f46" path="/var/lib/kubelet/pods/a6d28eec-663d-4019-b9d2-b8ea9dec9f46/volumes"
Nov 01 20:13:38.027611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:38.027571    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"]
Nov 01 20:13:38.033156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:38.033129    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefc2af9-n6brt"]
Nov 01 20:13:38.428570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:38.428536    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="019cadb2-8af5-43b7-8854-bfcb47ef571f" path="/var/lib/kubelet/pods/019cadb2-8af5-43b7-8854-bfcb47ef571f/volumes"
Nov 01 20:13:39.298469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.298428    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"]
Nov 01 20:13:39.306311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.306286    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.310712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.310684    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:13:39.311442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.311412    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"]
Nov 01 20:13:39.311931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.311698    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:13:39.311931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.311841    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:13:39.311931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.311922    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:13:39.474912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.474884    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkndl\" (UniqueName: \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.475241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.475217    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.475413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.475400    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.578103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.577929    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.578296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.578138    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkndl\" (UniqueName: \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.578296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.578235    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.583205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.583171    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.597113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.595112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkndl\" (UniqueName: \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.612798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.597745    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:39.630495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:39.629174    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:40.420831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:40.419822    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"]
Nov 01 20:13:41.278858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:41.277413    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh" event={"ID":"d09479a4-f95f-4cc9-83eb-156b8c967629","Type":"ContainerStarted","Data":"be16ce89683719de23d3b2d8efa148406e99c7edd8c392afe40ed1f486516b99"}
Nov 01 20:13:41.278858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:41.277483    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh" event={"ID":"d09479a4-f95f-4cc9-83eb-156b8c967629","Type":"ContainerStarted","Data":"a0b28116ec211350eedfededb0c93d2064c0793090d096872c6b6838b6daf40a"}
Nov 01 20:13:41.396046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:41.395963    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh" podStartSLOduration=2.395928853 podStartE2EDuration="2.395928853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:41.352431155 +0000 UTC m=+909.073509859" watchObservedRunningTime="2026-11-01 20:13:41.395928853 +0000 UTC m=+909.117007531"
Nov 01 20:13:42.289228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:42.289197    5041 generic.go:356] "Generic (PLEG): container finished" podID="d09479a4-f95f-4cc9-83eb-156b8c967629" containerID="be16ce89683719de23d3b2d8efa148406e99c7edd8c392afe40ed1f486516b99" exitCode=0
Nov 01 20:13:42.289619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:42.289287    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh" event={"ID":"d09479a4-f95f-4cc9-83eb-156b8c967629","Type":"ContainerDied","Data":"be16ce89683719de23d3b2d8efa148406e99c7edd8c392afe40ed1f486516b99"}
Nov 01 20:13:43.524116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.524082    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:43.611197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.609850    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\") pod \"d09479a4-f95f-4cc9-83eb-156b8c967629\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") "
Nov 01 20:13:43.611197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.609923    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\") pod \"d09479a4-f95f-4cc9-83eb-156b8c967629\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") "
Nov 01 20:13:43.611197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.610034    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\" (UniqueName: \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\") pod \"d09479a4-f95f-4cc9-83eb-156b8c967629\" (UID: \"d09479a4-f95f-4cc9-83eb-156b8c967629\") "
Nov 01 20:13:43.612339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.612311    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca" pod "d09479a4-f95f-4cc9-83eb-156b8c967629" (UID: "d09479a4-f95f-4cc9-83eb-156b8c967629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:43.615162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.615118    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl" pod "d09479a4-f95f-4cc9-83eb-156b8c967629" (UID: "d09479a4-f95f-4cc9-83eb-156b8c967629"). InnerVolumeSpecName "kube-api-access-dkndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:43.616706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.616667    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca" pod "d09479a4-f95f-4cc9-83eb-156b8c967629" (UID: "d09479a4-f95f-4cc9-83eb-156b8c967629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:13:43.711902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.711861    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:43.711902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.711895    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d09479a4-f95f-4cc9-83eb-156b8c967629-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:43.711902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:43.711908    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkndl\" (UniqueName: \"kubernetes.io/projected/d09479a4-f95f-4cc9-83eb-156b8c967629-kube-api-access-dkndl\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:44.323140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:44.323112    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"
Nov 01 20:13:44.323314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:44.323148    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh" event={"ID":"d09479a4-f95f-4cc9-83eb-156b8c967629","Type":"ContainerDied","Data":"a0b28116ec211350eedfededb0c93d2064c0793090d096872c6b6838b6daf40a"}
Nov 01 20:13:44.323314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:44.323203    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b28116ec211350eedfededb0c93d2064c0793090d096872c6b6838b6daf40a"
Nov 01 20:13:45.032472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:45.032436    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"]
Nov 01 20:13:45.040127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:45.040099    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8fb994-77wzp"]
Nov 01 20:13:46.024026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:46.023990    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"]
Nov 01 20:13:46.030380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:46.030345    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89edd9fb-5mnxz"]
Nov 01 20:13:46.429184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:46.429151    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f253dd7-9dc3-416b-a14f-730c2f5fc993" path="/var/lib/kubelet/pods/9f253dd7-9dc3-416b-a14f-730c2f5fc993/volumes"
Nov 01 20:13:46.429616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:46.429601    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7f63ed-4f98-43e9-9fac-728c621d723a" path="/var/lib/kubelet/pods/cf7f63ed-4f98-43e9-9fac-728c621d723a/volumes"
Nov 01 20:13:48.368591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.368509    5041 generic.go:356] "Generic (PLEG): container finished" podID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerID="b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e" exitCode=0
Nov 01 20:13:48.369031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.368586    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerDied","Data":"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"}
Nov 01 20:13:48.369122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.369107    5041 scope.go:122] "RemoveContainer" containerID="b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"
Nov 01 20:13:48.370474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.370446    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:48.370540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.370507    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:48.370834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:48.370537    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:49.394809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:49.387272    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerStarted","Data":"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"}
Nov 01 20:13:49.590485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:49.590448    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:49.590947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:49.590890    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:49.591119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:49.591013    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:50.247531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.244631    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"]
Nov 01 20:13:50.247531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.245702    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d09479a4-f95f-4cc9-83eb-156b8c967629" containerName="revoke-keys"
Nov 01 20:13:50.247531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.245726    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09479a4-f95f-4cc9-83eb-156b8c967629" containerName="revoke-keys"
Nov 01 20:13:50.247531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.245993    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09479a4-f95f-4cc9-83eb-156b8c967629" containerName="revoke-keys"
Nov 01 20:13:50.254477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.250811    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.283658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.277296    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:13:50.283658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.277393    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:13:50.283658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.277422    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:13:50.283658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.277485    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:13:50.317869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.306568    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"]
Nov 01 20:13:50.325818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.323861    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.325818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.323928    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddb7\" (UniqueName: \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.325818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.324009    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.425811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.425738    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.426277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.425831    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddb7\" (UniqueName: \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.426277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.425906    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.426657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.426633    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.428335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.428303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.433891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.433867    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddb7\" (UniqueName: \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\") pod \"maas-api-revoke-sub-6bebfd38-zg6jc\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:50.591957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.591924    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:50.592237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.592210    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:50.642028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:50.641995    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:51.022521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:51.022487    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"]
Nov 01 20:13:51.023051 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:13:51.023029    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd07240_059e_4623_b259_5eb0efe8e43f.slice/crio-931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e WatchSource:0}: Error finding container 931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e: Status 404 returned error can't find the container with id 931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e
Nov 01 20:13:51.439578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:51.439497    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc" event={"ID":"8bd07240-059e-4623-b259-5eb0efe8e43f","Type":"ContainerStarted","Data":"931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e"}
Nov 01 20:13:51.590570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:51.590178    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:51.590570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:51.590234    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:52.029714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.029629    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"]
Nov 01 20:13:52.035808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.035750    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b9bb640-jv8j4"]
Nov 01 20:13:52.431758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.431719    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b4899d-9a54-480e-ad2f-55f7714adb25" path="/var/lib/kubelet/pods/96b4899d-9a54-480e-ad2f-55f7714adb25/volumes"
Nov 01 20:13:52.478282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.478219    5041 generic.go:356] "Generic (PLEG): container finished" podID="8bd07240-059e-4623-b259-5eb0efe8e43f" containerID="de4f593ac7be33ca239cd3b55ee311d4d776e40a327d03a7a7533b41d266c879" exitCode=0
Nov 01 20:13:52.478671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.478302    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc" event={"ID":"8bd07240-059e-4623-b259-5eb0efe8e43f","Type":"ContainerDied","Data":"de4f593ac7be33ca239cd3b55ee311d4d776e40a327d03a7a7533b41d266c879"}
Nov 01 20:13:52.591023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.590973    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:52.591194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:52.591056    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:53.590610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.590147    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:53.590610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.590213    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:53.761854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.740906    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:53.836800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.836497    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\") pod \"8bd07240-059e-4623-b259-5eb0efe8e43f\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") "
Nov 01 20:13:53.836800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.836678    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\") pod \"8bd07240-059e-4623-b259-5eb0efe8e43f\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") "
Nov 01 20:13:53.836800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.836756    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\" (UniqueName: \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\") pod \"8bd07240-059e-4623-b259-5eb0efe8e43f\" (UID: \"8bd07240-059e-4623-b259-5eb0efe8e43f\") "
Nov 01 20:13:53.838276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.838165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca" pod "8bd07240-059e-4623-b259-5eb0efe8e43f" (UID: "8bd07240-059e-4623-b259-5eb0efe8e43f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:13:53.840762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.840708    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca" pod "8bd07240-059e-4623-b259-5eb0efe8e43f" (UID: "8bd07240-059e-4623-b259-5eb0efe8e43f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:13:53.841017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.840767    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7" pod "8bd07240-059e-4623-b259-5eb0efe8e43f" (UID: "8bd07240-059e-4623-b259-5eb0efe8e43f"). InnerVolumeSpecName "kube-api-access-dddb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:13:53.938478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.938442    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:53.938478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.938480    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddb7\" (UniqueName: \"kubernetes.io/projected/8bd07240-059e-4623-b259-5eb0efe8e43f-kube-api-access-dddb7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:53.938720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:53.938498    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd07240-059e-4623-b259-5eb0efe8e43f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:13:54.498885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.498849    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"
Nov 01 20:13:54.499043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.498892    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc" event={"ID":"8bd07240-059e-4623-b259-5eb0efe8e43f","Type":"ContainerDied","Data":"931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e"}
Nov 01 20:13:54.499043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.498940    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="931e2a2b549816a31a8001e52d50539ff91308683c4ef5e4061b2767fb61ca0e"
Nov 01 20:13:54.592107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.591730    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body=
Nov 01 20:13:54.592107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.591841    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused"
Nov 01 20:13:54.822364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.822317    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"]
Nov 01 20:13:54.833027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:54.832905    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ebb0cff-75jnh"]
Nov 01 20:13:55.611046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:55.611005    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:55.612012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:55.611989    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:55.613933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:55.613732    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:13:56.304474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.304436    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:13:56.305560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.305537    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8bd07240-059e-4623-b259-5eb0efe8e43f" containerName="revoke-keys"
Nov 01 20:13:56.305687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.305563    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd07240-059e-4623-b259-5eb0efe8e43f" containerName="revoke-keys"
Nov 01 20:13:56.305810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.305794    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd07240-059e-4623-b259-5eb0efe8e43f" containerName="revoke-keys"
Nov 01 20:13:56.310848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.310822    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.331716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.331680    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:13:56.342192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.342167    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-91f1523a-gw-openshift-default-dockercfg-rltn7\""
Nov 01 20:13:56.429950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.429916    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09479a4-f95f-4cc9-83eb-156b8c967629" path="/var/lib/kubelet/pods/d09479a4-f95f-4cc9-83eb-156b8c967629/volumes"
Nov 01 20:13:56.441750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.441721    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.441934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.441760    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.441934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.441866    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtfq\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.441940    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.442054    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.442103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.442230    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.442285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.442339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.442325    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.555918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.555995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556070    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtfq\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556199    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556255    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556287    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556383    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556458    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.556486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.557052    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.557314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.559237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.557921    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.573426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.571528    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.573426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.571961    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.597574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.597502    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.611816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.607356    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.615813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.614462    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.615813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.615147    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtfq\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\") pod \"e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") " pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.646081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.646045    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:56.988228 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:13:56.988195    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb224fe8_7f16_43db_b4e4_1f2d910bbb91.slice/crio-68c05a6efb66137015f57142d9272696fd3cee798e37cd7df5cdeec5bfbaea59 WatchSource:0}: Error finding container 68c05a6efb66137015f57142d9272696fd3cee798e37cd7df5cdeec5bfbaea59: Status 404 returned error can't find the container with id 68c05a6efb66137015f57142d9272696fd3cee798e37cd7df5cdeec5bfbaea59
Nov 01 20:13:56.989045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.988915    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:13:56.992317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.992283    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:56.992437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.992367    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:56.992437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:56.992418    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:13:57.025650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.025621    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"]
Nov 01 20:13:57.029663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.029641    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"]
Nov 01 20:13:57.034541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.034515    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acf3a5a-2s62j"]
Nov 01 20:13:57.039708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.039684    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05deae6b-bw6cb"]
Nov 01 20:13:57.540030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.539975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerStarted","Data":"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"}
Nov 01 20:13:57.540219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.540055    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerStarted","Data":"68c05a6efb66137015f57142d9272696fd3cee798e37cd7df5cdeec5bfbaea59"}
Nov 01 20:13:57.561660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.560036    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" podStartSLOduration=1.56001667 podStartE2EDuration="1.56001667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:57.556622246 +0000 UTC m=+925.277700919" watchObservedRunningTime="2026-11-01 20:13:57.56001667 +0000 UTC m=+925.281095346"
Nov 01 20:13:57.646606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:57.646575    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:58.429599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:58.429557    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e29abde-1803-404c-ad0e-d7c8884f8946" path="/var/lib/kubelet/pods/9e29abde-1803-404c-ad0e-d7c8884f8946/volumes"
Nov 01 20:13:58.430129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:58.430108    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc328c37-b985-4099-8ca8-19b753a53b3a" path="/var/lib/kubelet/pods/bc328c37-b985-4099-8ca8-19b753a53b3a/volumes"
Nov 01 20:13:58.541902 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:13:58.541863    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:13:58.652144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:58.652116    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:59.561187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:59.561153    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:13:59.562444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:13:59.562420    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:14:02.024195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:02.024157    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"]
Nov 01 20:14:02.029865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:02.029832    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8854ed13-ffdbw"]
Nov 01 20:14:02.430382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:02.430351    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1869d6-4648-498c-a591-cb2e14fb49b0" path="/var/lib/kubelet/pods/5e1869d6-4648-498c-a591-cb2e14fb49b0/volumes"
Nov 01 20:14:08.779641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:08.779605    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:14:08.780436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:08.780402    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy" containerID="cri-o://c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761" gracePeriod=30
Nov 01 20:14:11.025380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:11.025343    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"]
Nov 01 20:14:11.030685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:11.030658    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019514d2-5qtxf"]
Nov 01 20:14:12.442726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:12.442689    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab8cb35-3d15-43ab-9a7e-a1dc6317d335" path="/var/lib/kubelet/pods/0ab8cb35-3d15-43ab-9a7e-a1dc6317d335/volumes"
Nov 01 20:14:13.026011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:13.025923    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"]
Nov 01 20:14:13.033491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:13.033456    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7129d2a5-4c268"]
Nov 01 20:14:14.061351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.061284    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:14:14.173156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173118    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173198    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173316    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173401    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173539    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173610    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.173674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.173658    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.174854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.174523    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\") pod \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\" (UID: \"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f\") "
Nov 01 20:14:14.175378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.175009    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.175378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.175262    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.175853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.175831    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.175968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.175942    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"]
Nov 01 20:14:14.176235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.176216    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.176511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.176495    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.176909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.176843    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:14.177016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.176936    5041 cpu_manager.go:414] "containerMap: removing container" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy"
Nov 01 20:14:14.177016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.176953    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy"
Nov 01 20:14:14.177155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.177141    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerName="istio-proxy"
Nov 01 20:14:14.207159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.206609    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:14:14.207159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.206764    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:14.212749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.211619    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "kube-api-access-gwklg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:14.237860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.213627    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token" pod "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" (UID: "eebaffa7-2228-4bd7-81d5-a45aa8e8a78f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:14.291070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291034    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291067    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291083    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291093    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291109    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291119    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwklg\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-kube-api-access-gwklg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291133    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.291313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.291142    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:14.300864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.296915    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.300864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.299904    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"]
Nov 01 20:14:14.302977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.302954    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:14:14.303447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.302989    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:14:14.303534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.303198    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:14:14.303683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.303656    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:14:14.392124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.392021    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.392124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.392069    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29tj\" (UniqueName: \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.392317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.392284    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.429534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.429499    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd0c044-8ad7-42a2-9275-fd7c412f5716" path="/var/lib/kubelet/pods/0cd0c044-8ad7-42a2-9275-fd7c412f5716/volumes"
Nov 01 20:14:14.493949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.493908    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.494157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.494023    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.494157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.494043    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29tj\" (UniqueName: \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.494700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.494676    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.496499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.496474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.501671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.501650    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29tj\" (UniqueName: \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\") pod \"maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.614558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.614516    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:14.741728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.741688    5041 generic.go:356] "Generic (PLEG): container finished" podID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" containerID="c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761" exitCode=0
Nov 01 20:14:14.741954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.741877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" event={"ID":"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f","Type":"ContainerDied","Data":"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"}
Nov 01 20:14:14.741954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.741931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw" event={"ID":"eebaffa7-2228-4bd7-81d5-a45aa8e8a78f","Type":"ContainerDied","Data":"dbe8aae9cb0331283bcbfbf7feec8c76dfd9c85d200749e5df481a951b7019a1"}
Nov 01 20:14:14.742081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.741961    5041 scope.go:122] "RemoveContainer" containerID="c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"
Nov 01 20:14:14.762805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.742151    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"
Nov 01 20:14:14.788535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.788411    5041 scope.go:122] "RemoveContainer" containerID="c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"
Nov 01 20:14:14.788838 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:14:14.788754    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761\": container with ID starting with c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761 not found: ID does not exist" containerID="c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"
Nov 01 20:14:14.788838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.788808    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761"} err="failed to get container status \"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761\": rpc error: code = NotFound desc = could not find container \"c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761\": container with ID starting with c90ed3ee513e3a8ebf8338520a597e7aef54243845e578a1a1aa30904ca27761 not found: ID does not exist"
Nov 01 20:14:14.795183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.795161    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:14:14.800172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.800148    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-638f19-openshift-default-544bc89849-psmmw"]
Nov 01 20:14:14.930026 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:14:14.929995    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75edde5_8cc6_4c7f_8963_2963521d2d95.slice/crio-c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e WatchSource:0}: Error finding container c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e: Status 404 returned error can't find the container with id c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e
Nov 01 20:14:14.931953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:14.931929    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"]
Nov 01 20:14:15.779607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:15.779517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x" event={"ID":"c75edde5-8cc6-4c7f-8963-2963521d2d95","Type":"ContainerStarted","Data":"b8b4fcad66772ed02bd69cef3614940e352a893a88e677b32070432cc982ef24"}
Nov 01 20:14:15.779607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:15.779571    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x" event={"ID":"c75edde5-8cc6-4c7f-8963-2963521d2d95","Type":"ContainerStarted","Data":"c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e"}
Nov 01 20:14:15.798691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:15.798632    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x" podStartSLOduration=1.7986176390000002 podStartE2EDuration="1.798617639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:15.794212409 +0000 UTC m=+943.515291088" watchObservedRunningTime="2026-11-01 20:14:15.798617639 +0000 UTC m=+943.519696314"
Nov 01 20:14:16.024597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.024565    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"]
Nov 01 20:14:16.029435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.029407    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f77b5ab-9fbbk"]
Nov 01 20:14:16.429216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.429127    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e61d48-ccb9-435a-8970-8e9a7b611aa4" path="/var/lib/kubelet/pods/57e61d48-ccb9-435a-8970-8e9a7b611aa4/volumes"
Nov 01 20:14:16.429637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.429622    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eebaffa7-2228-4bd7-81d5-a45aa8e8a78f" path="/var/lib/kubelet/pods/eebaffa7-2228-4bd7-81d5-a45aa8e8a78f/volumes"
Nov 01 20:14:16.790046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.790011    5041 generic.go:356] "Generic (PLEG): container finished" podID="c75edde5-8cc6-4c7f-8963-2963521d2d95" containerID="b8b4fcad66772ed02bd69cef3614940e352a893a88e677b32070432cc982ef24" exitCode=0
Nov 01 20:14:16.790470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:16.790088    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x" event={"ID":"c75edde5-8cc6-4c7f-8963-2963521d2d95","Type":"ContainerDied","Data":"b8b4fcad66772ed02bd69cef3614940e352a893a88e677b32070432cc982ef24"}
Nov 01 20:14:18.012097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.012071    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:18.171276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.171193    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\") pod \"c75edde5-8cc6-4c7f-8963-2963521d2d95\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") "
Nov 01 20:14:18.171276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.171238    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\") pod \"c75edde5-8cc6-4c7f-8963-2963521d2d95\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") "
Nov 01 20:14:18.171484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.171306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\" (UniqueName: \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\") pod \"c75edde5-8cc6-4c7f-8963-2963521d2d95\" (UID: \"c75edde5-8cc6-4c7f-8963-2963521d2d95\") "
Nov 01 20:14:18.171616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.171589    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca" pod "c75edde5-8cc6-4c7f-8963-2963521d2d95" (UID: "c75edde5-8cc6-4c7f-8963-2963521d2d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:18.171835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.171811    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:18.173545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.173518    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca" pod "c75edde5-8cc6-4c7f-8963-2963521d2d95" (UID: "c75edde5-8cc6-4c7f-8963-2963521d2d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:18.173640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.173579    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj" pod "c75edde5-8cc6-4c7f-8963-2963521d2d95" (UID: "c75edde5-8cc6-4c7f-8963-2963521d2d95"). InnerVolumeSpecName "kube-api-access-d29tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:18.277650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.277609    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75edde5-8cc6-4c7f-8963-2963521d2d95-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:18.277650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.277652    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29tj\" (UniqueName: \"kubernetes.io/projected/c75edde5-8cc6-4c7f-8963-2963521d2d95-kube-api-access-d29tj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:18.814582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.814547    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"
Nov 01 20:14:18.814582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.814575    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x" event={"ID":"c75edde5-8cc6-4c7f-8963-2963521d2d95","Type":"ContainerDied","Data":"c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e"}
Nov 01 20:14:18.814871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:18.814627    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3f00c90862dfde5e9dcc27863ca017a742a91e7dfe53f08347483c1d51a1a7e"
Nov 01 20:14:21.024099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:21.024052    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"]
Nov 01 20:14:21.029153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:21.029123    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4bd439f-bnxgf"]
Nov 01 20:14:22.429463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:22.429422    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ec6baa-35a6-4538-bfbd-833185672008" path="/var/lib/kubelet/pods/86ec6baa-35a6-4538-bfbd-833185672008/volumes"
Nov 01 20:14:23.026136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:23.026093    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"]
Nov 01 20:14:23.034168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:23.034133    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d85f7a5-lqpr9"]
Nov 01 20:14:24.429298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.429264    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e272e37a-f4ca-428e-b9c7-f381eb55b913" path="/var/lib/kubelet/pods/e272e37a-f4ca-428e-b9c7-f381eb55b913/volumes"
Nov 01 20:14:24.891808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.891748    5041 generic.go:356] "Generic (PLEG): container finished" podID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerID="2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f" exitCode=0
Nov 01 20:14:24.892072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.891810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerDied","Data":"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"}
Nov 01 20:14:24.892284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.892269    5041 scope.go:122] "RemoveContainer" containerID="2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"
Nov 01 20:14:24.893751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.893724    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:14:24.893853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.893812    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:14:24.893853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:24.893847    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:14:25.915900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:25.915840    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerStarted","Data":"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"}
Nov 01 20:14:26.574120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.574078    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"]
Nov 01 20:14:26.582521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.582485    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-91f1523a-kjq2x"]
Nov 01 20:14:26.661803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.658155    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:14:26.665800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.664908    5041 patch_prober.go:29] interesting pod/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 20:14:26.665800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.665007    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 20:14:26.725804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.724768    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:14:26.991862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:26.991055    5041 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" secret="" err="secret \"e2e-migrate-91f1523a-gw-openshift-default-dockercfg-rltn7\" not found"
Nov 01 20:14:27.060803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:27.056659    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"]
Nov 01 20:14:27.092192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:27.092009    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d280331d-4fthw"]
Nov 01 20:14:27.665716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:27.654274    5041 patch_prober.go:29] interesting pod/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 01 20:14:27.665716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:27.654349    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 01 20:14:28.014859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:28.008994    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy" containerID="cri-o://3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da" gracePeriod=30
Nov 01 20:14:28.429706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:28.429671    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ee9b06-9cd4-4227-bbfa-ec722ba76d3c" path="/var/lib/kubelet/pods/81ee9b06-9cd4-4227-bbfa-ec722ba76d3c/volumes"
Nov 01 20:14:28.430208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:28.430193    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75edde5-8cc6-4c7f-8963-2963521d2d95" path="/var/lib/kubelet/pods/c75edde5-8cc6-4c7f-8963-2963521d2d95/volumes"
Nov 01 20:14:28.628575 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:14:28.628535    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:14:29.566957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:29.566920    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:14:33.361485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.361457    5041 scope.go:122] "RemoveContainer" containerID="1c1e73c6227dedc214a55fa457784f62e2978214ee664d8795ae461c4e7777cc"
Nov 01 20:14:33.372852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.372774    5041 scope.go:122] "RemoveContainer" containerID="51b66fed496f0b5bca8566df493e36e99ef91676e5e43a59caa3fffccf14af2d"
Nov 01 20:14:33.383154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.383130    5041 scope.go:122] "RemoveContainer" containerID="5deca8d0c59e2dd758785842a4486987894b844449fec3b200f8964de92d475b"
Nov 01 20:14:33.394037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.394014    5041 scope.go:122] "RemoveContainer" containerID="b422390e6059ab49eade7f9e861645dbac8d7841c5feaea86d4369c2fce903b0"
Nov 01 20:14:33.405242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.405215    5041 scope.go:122] "RemoveContainer" containerID="3ce99d9c5a5b7bbb20236ea6162fde975350586d71e02ff153599d9546a721b0"
Nov 01 20:14:33.416629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.416610    5041 scope.go:122] "RemoveContainer" containerID="f5ee05f93eed72cf428ff2bd60218031efa87f2974caaf117fd275dfa61fbb53"
Nov 01 20:14:33.428001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.427981    5041 scope.go:122] "RemoveContainer" containerID="d6b463c3afc27ab83502299a0ed1283b93ed9e0fc87de705e91c5ed729400cae"
Nov 01 20:14:33.438866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.438844    5041 scope.go:122] "RemoveContainer" containerID="e835c1d67208b9d9b4d2c5f65d36bfe3dfd980bcf87fb538a4d37bafd04e95b7"
Nov 01 20:14:33.453814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:33.453743    5041 scope.go:122] "RemoveContainer" containerID="4b7a147060d9743e4b7a25b2d6748550e4cdf4384c69f97a8930da286f52b4f5"
Nov 01 20:14:36.613461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.613433    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:14:36.752502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752414    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752480    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752518    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752572    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752596    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752678    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752717    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752719    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:36.752751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752749    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\") pod \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\" (UID: \"fb224fe8-7f16-43db-b4e4-1f2d910bbb91\") "
Nov 01 20:14:36.753219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:36.753219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752975    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:36.753219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.752990    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:36.753375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753314    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:36.753530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753510    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.753530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753529    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.753671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753538    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.753671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753547    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.753671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.753556    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.755008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.754970    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:36.755114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.755051    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "kube-api-access-xhtfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:36.755503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.755471    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:14:36.755593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.755513    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy" pod "fb224fe8-7f16-43db-b4e4-1f2d910bbb91" (UID: "fb224fe8-7f16-43db-b4e4-1f2d910bbb91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:14:36.854996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.854908    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.854996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.854941    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtfq\" (UniqueName: \"kubernetes.io/projected/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-kube-api-access-xhtfq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.854996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.854959    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:36.854996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:36.854970    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb224fe8-7f16-43db-b4e4-1f2d910bbb91-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:37.110462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.110423    5041 generic.go:356] "Generic (PLEG): container finished" podID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerID="3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da" exitCode=0
Nov 01 20:14:37.110639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.110490    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"
Nov 01 20:14:37.110639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.110498    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerDied","Data":"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"}
Nov 01 20:14:37.110639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.110580    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq" event={"ID":"fb224fe8-7f16-43db-b4e4-1f2d910bbb91","Type":"ContainerDied","Data":"68c05a6efb66137015f57142d9272696fd3cee798e37cd7df5cdeec5bfbaea59"}
Nov 01 20:14:37.110639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.110604    5041 scope.go:122] "RemoveContainer" containerID="3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"
Nov 01 20:14:37.123433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.123412    5041 scope.go:122] "RemoveContainer" containerID="2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"
Nov 01 20:14:37.135632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.135607    5041 scope.go:122] "RemoveContainer" containerID="3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"
Nov 01 20:14:37.136003 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:14:37.135978    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da\": container with ID starting with 3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da not found: ID does not exist" containerID="3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"
Nov 01 20:14:37.136120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.136015    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da"} err="failed to get container status \"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da\": rpc error: code = NotFound desc = could not find container \"3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da\": container with ID starting with 3ab23008245e839b5720882cd545449a2005dc76ec27654b4442a32bad43c4da not found: ID does not exist"
Nov 01 20:14:37.136120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.136039    5041 scope.go:122] "RemoveContainer" containerID="2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"
Nov 01 20:14:37.136360 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:14:37.136331    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f\": container with ID starting with 2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f not found: ID does not exist" containerID="2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"
Nov 01 20:14:37.136449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.136366    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f"} err="failed to get container status \"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f\": rpc error: code = NotFound desc = could not find container \"2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f\": container with ID starting with 2d01466a1a3d093f5395d11c680b1ba5e3f41efb8d7d8940e1f0db0be868d55f not found: ID does not exist"
Nov 01 20:14:37.139950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.139927    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:14:37.144720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:37.144699    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-91f1523a-gw-openshift-default-64b5bfb9d9-jkqtq"]
Nov 01 20:14:38.429337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:38.429300    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" path="/var/lib/kubelet/pods/fb224fe8-7f16-43db-b4e4-1f2d910bbb91/volumes"
Nov 01 20:14:41.022439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:41.022403    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"]
Nov 01 20:14:41.026907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:41.026886    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b762708d-rqddq"]
Nov 01 20:14:42.429163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:42.429133    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c800e33c-4ef5-4f98-abaa-48ff15bd6ee0" path="/var/lib/kubelet/pods/c800e33c-4ef5-4f98-abaa-48ff15bd6ee0/volumes"
Nov 01 20:14:50.032196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.032161    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"]
Nov 01 20:14:50.039492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.039467    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"]
Nov 01 20:14:50.044748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.044723    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94716fca-54ddk"]
Nov 01 20:14:50.048698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.048664    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5362b1e-pmqvn"]
Nov 01 20:14:50.428658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.428624    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b" path="/var/lib/kubelet/pods/45e35dc3-6727-4aaa-bde6-e1a4f2d9d55b/volumes"
Nov 01 20:14:50.429147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:50.429131    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f482259a-049f-4625-ba3a-70223cc5852e" path="/var/lib/kubelet/pods/f482259a-049f-4625-ba3a-70223cc5852e/volumes"
Nov 01 20:14:52.606870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.606829    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"]
Nov 01 20:14:52.608583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.608551    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:52.608962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.608941    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:52.614872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.609136    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c75edde5-8cc6-4c7f-8963-2963521d2d95" containerName="revoke-keys"
Nov 01 20:14:52.615071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.615049    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75edde5-8cc6-4c7f-8963-2963521d2d95" containerName="revoke-keys"
Nov 01 20:14:52.615424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.615405    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:52.615551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.615540    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:52.615642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.615632    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75edde5-8cc6-4c7f-8963-2963521d2d95" containerName="revoke-keys"
Nov 01 20:14:52.634032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.634000    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"]
Nov 01 20:14:52.634388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.634318    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.647240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.647208    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:14:52.647712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.647691    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:14:52.650503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.647974    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:14:52.656843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.652948    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:14:52.679828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.679746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.680272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.680249    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfvh\" (UniqueName: \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.680474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.680455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.782045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.782010    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.782218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.782148    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfvh\" (UniqueName: \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.782218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.782174    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.782737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.782703    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.784569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.784543    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.796498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.796470    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfvh\" (UniqueName: \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\") pod \"maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:52.993349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:52.993260    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:53.286198 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:14:53.286158    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab900363_9fbb_41fe_8e89_b27501cb7f41.slice/crio-bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901 WatchSource:0}: Error finding container bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901: Status 404 returned error can't find the container with id bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901
Nov 01 20:14:53.287862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.287845    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"]
Nov 01 20:14:53.598932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.598881    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"]
Nov 01 20:14:53.599927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.599898    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:53.599927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.599921    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb224fe8-7f16-43db-b4e4-1f2d910bbb91" containerName="istio-proxy"
Nov 01 20:14:53.610733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.610695    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.620291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.619944    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"]
Nov 01 20:14:53.698539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.698154    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.698539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.698218    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4668\" (UniqueName: \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.698539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.698279    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.800035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.799995    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.800234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.800060    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4668\" (UniqueName: \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.800234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.800147    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.801272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.801219    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.803564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.803540    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.808566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.808537    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4668\" (UniqueName: \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\") pod \"maas-api-revoke-sub-282dc1e6-rwnp8\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:53.958022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:53.957921    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:54.324481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:54.318759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb" event={"ID":"ab900363-9fbb-41fe-8e89-b27501cb7f41","Type":"ContainerStarted","Data":"97d446a82b58cc3f2285c5d465fe5206d449843655c143ce9481b6eb3d16003b"}
Nov 01 20:14:54.324481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:54.318868    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb" event={"ID":"ab900363-9fbb-41fe-8e89-b27501cb7f41","Type":"ContainerStarted","Data":"bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901"}
Nov 01 20:14:54.329753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:54.328322    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"]
Nov 01 20:14:54.347187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:54.347125    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb" podStartSLOduration=2.34710733 podStartE2EDuration="2.34710733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:54.346064663 +0000 UTC m=+982.067143340" watchObservedRunningTime="2026-11-01 20:14:54.34710733 +0000 UTC m=+982.068186007"
Nov 01 20:14:55.334802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:55.285839    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"]
Nov 01 20:14:55.964845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:55.964488    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.027815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.026273    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"]
Nov 01 20:14:56.162803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.155691    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.162803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.155830    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.162803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.156097    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl6qv\" (UniqueName: \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.258861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.258664    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl6qv\" (UniqueName: \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.258861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.258832    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.259111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.258909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.280248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.280218    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.281007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.280977    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.288540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.288487    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl6qv\" (UniqueName: \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\") pod \"maas-api-revoke-sub-d0b6f9dc-8cqhh\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.319802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.301328    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:56.319802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.316030    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8" event={"ID":"bc243873-82c7-43d8-8851-1334401203b2","Type":"ContainerStarted","Data":"9903b7d31b9ae4f4dbfeb1ead972930bb15b996c4ae260e36cf9c70937701971"}
Nov 01 20:14:56.321058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.320368    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8" event={"ID":"bc243873-82c7-43d8-8851-1334401203b2","Type":"ContainerStarted","Data":"eb0b333c72fcc13394f2c5fa926575b30129db0ce56ff8c281cdc9e030c1b81c"}
Nov 01 20:14:56.496803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.490684    5041 generic.go:356] "Generic (PLEG): container finished" podID="ab900363-9fbb-41fe-8e89-b27501cb7f41" containerID="97d446a82b58cc3f2285c5d465fe5206d449843655c143ce9481b6eb3d16003b" exitCode=0
Nov 01 20:14:56.503805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:56.501845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb" event={"ID":"ab900363-9fbb-41fe-8e89-b27501cb7f41","Type":"ContainerDied","Data":"97d446a82b58cc3f2285c5d465fe5206d449843655c143ce9481b6eb3d16003b"}
Nov 01 20:14:57.032946 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:14:57.032919    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83737883_64a0_4dcb_b495_6d5ffd6d0a1c.slice/crio-cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20 WatchSource:0}: Error finding container cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20: Status 404 returned error can't find the container with id cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20
Nov 01 20:14:57.035327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.035301    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"]
Nov 01 20:14:57.502120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.502084    5041 generic.go:356] "Generic (PLEG): container finished" podID="bc243873-82c7-43d8-8851-1334401203b2" containerID="9903b7d31b9ae4f4dbfeb1ead972930bb15b996c4ae260e36cf9c70937701971" exitCode=0
Nov 01 20:14:57.502561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.502153    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8" event={"ID":"bc243873-82c7-43d8-8851-1334401203b2","Type":"ContainerDied","Data":"9903b7d31b9ae4f4dbfeb1ead972930bb15b996c4ae260e36cf9c70937701971"}
Nov 01 20:14:57.504120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.504088    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh" event={"ID":"83737883-64a0-4dcb-b495-6d5ffd6d0a1c","Type":"ContainerStarted","Data":"3d0cee4f0179816be04f787f47c905dbe8d5ae8a52178215d618bb60c80f6d25"}
Nov 01 20:14:57.504255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.504136    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh" event={"ID":"83737883-64a0-4dcb-b495-6d5ffd6d0a1c","Type":"ContainerStarted","Data":"cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20"}
Nov 01 20:14:57.529195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.529131    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh" podStartSLOduration=2.529113792 podStartE2EDuration="2.529113792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:14:57.523667535 +0000 UTC m=+985.244746212" watchObservedRunningTime="2026-11-01 20:14:57.529113792 +0000 UTC m=+985.250192514"
Nov 01 20:14:57.695388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.695358    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:57.735271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.735238    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:57.805588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805554    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\") pod \"bc243873-82c7-43d8-8851-1334401203b2\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") "
Nov 01 20:14:57.805588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805590    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\" (UniqueName: \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\") pod \"bc243873-82c7-43d8-8851-1334401203b2\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") "
Nov 01 20:14:57.805859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805632    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\") pod \"ab900363-9fbb-41fe-8e89-b27501cb7f41\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") "
Nov 01 20:14:57.805859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805666    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\") pod \"bc243873-82c7-43d8-8851-1334401203b2\" (UID: \"bc243873-82c7-43d8-8851-1334401203b2\") "
Nov 01 20:14:57.805973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805868    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\" (UniqueName: \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\") pod \"ab900363-9fbb-41fe-8e89-b27501cb7f41\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") "
Nov 01 20:14:57.805973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805957    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\") pod \"ab900363-9fbb-41fe-8e89-b27501cb7f41\" (UID: \"ab900363-9fbb-41fe-8e89-b27501cb7f41\") "
Nov 01 20:14:57.806075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.805995    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca" pod "bc243873-82c7-43d8-8851-1334401203b2" (UID: "bc243873-82c7-43d8-8851-1334401203b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:57.806128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.806072    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca" pod "ab900363-9fbb-41fe-8e89-b27501cb7f41" (UID: "ab900363-9fbb-41fe-8e89-b27501cb7f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:57.806970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.806942    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc243873-82c7-43d8-8851-1334401203b2-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:57.806970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.806971    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:57.807998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.807974    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca" pod "bc243873-82c7-43d8-8851-1334401203b2" (UID: "bc243873-82c7-43d8-8851-1334401203b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:57.807998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.807987    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh" pod "ab900363-9fbb-41fe-8e89-b27501cb7f41" (UID: "ab900363-9fbb-41fe-8e89-b27501cb7f41"). InnerVolumeSpecName "kube-api-access-mtfvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:57.808398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.808382    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca" pod "ab900363-9fbb-41fe-8e89-b27501cb7f41" (UID: "ab900363-9fbb-41fe-8e89-b27501cb7f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:57.808398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.808391    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668" pod "bc243873-82c7-43d8-8851-1334401203b2" (UID: "bc243873-82c7-43d8-8851-1334401203b2"). InnerVolumeSpecName "kube-api-access-n4668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:57.908576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.908506    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4668\" (UniqueName: \"kubernetes.io/projected/bc243873-82c7-43d8-8851-1334401203b2-kube-api-access-n4668\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:57.908576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.908534    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc243873-82c7-43d8-8851-1334401203b2-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:57.908576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.908544    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfvh\" (UniqueName: \"kubernetes.io/projected/ab900363-9fbb-41fe-8e89-b27501cb7f41-kube-api-access-mtfvh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:57.908576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:57.908553    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab900363-9fbb-41fe-8e89-b27501cb7f41-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:58.515048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.515009    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8" event={"ID":"bc243873-82c7-43d8-8851-1334401203b2","Type":"ContainerDied","Data":"eb0b333c72fcc13394f2c5fa926575b30129db0ce56ff8c281cdc9e030c1b81c"}
Nov 01 20:14:58.515048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.515050    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"
Nov 01 20:14:58.515557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.515071    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0b333c72fcc13394f2c5fa926575b30129db0ce56ff8c281cdc9e030c1b81c"
Nov 01 20:14:58.517298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.517253    5041 generic.go:356] "Generic (PLEG): container finished" podID="83737883-64a0-4dcb-b495-6d5ffd6d0a1c" containerID="3d0cee4f0179816be04f787f47c905dbe8d5ae8a52178215d618bb60c80f6d25" exitCode=0
Nov 01 20:14:58.517598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.517575    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh" event={"ID":"83737883-64a0-4dcb-b495-6d5ffd6d0a1c","Type":"ContainerDied","Data":"3d0cee4f0179816be04f787f47c905dbe8d5ae8a52178215d618bb60c80f6d25"}
Nov 01 20:14:58.519341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.519317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb" event={"ID":"ab900363-9fbb-41fe-8e89-b27501cb7f41","Type":"ContainerDied","Data":"bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901"}
Nov 01 20:14:58.519431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.519353    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef38a707192aaf33d9dc989ceccd21ec0d4974cd0df9ae183cf1ae422f71901"
Nov 01 20:14:58.519431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:58.519325    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"
Nov 01 20:14:58.543026 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:14:58.542950    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:14:59.690168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.690145    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:14:59.833799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.833748    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\") pod \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") "
Nov 01 20:14:59.833970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.833846    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\") pod \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") "
Nov 01 20:14:59.834061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.833970    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\" (UniqueName: \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\") pod \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\" (UID: \"83737883-64a0-4dcb-b495-6d5ffd6d0a1c\") "
Nov 01 20:14:59.834124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.834056    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca" pod "83737883-64a0-4dcb-b495-6d5ffd6d0a1c" (UID: "83737883-64a0-4dcb-b495-6d5ffd6d0a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:14:59.834487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.834464    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:59.836028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.836005    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv" pod "83737883-64a0-4dcb-b495-6d5ffd6d0a1c" (UID: "83737883-64a0-4dcb-b495-6d5ffd6d0a1c"). InnerVolumeSpecName "kube-api-access-vl6qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:14:59.836103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.836003    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca" pod "83737883-64a0-4dcb-b495-6d5ffd6d0a1c" (UID: "83737883-64a0-4dcb-b495-6d5ffd6d0a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:14:59.935757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.935716    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:14:59.935757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:14:59.935749    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl6qv\" (UniqueName: \"kubernetes.io/projected/83737883-64a0-4dcb-b495-6d5ffd6d0a1c-kube-api-access-vl6qv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:00.142589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.142507    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:00.143433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143404    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ab900363-9fbb-41fe-8e89-b27501cb7f41" containerName="revoke-keys"
Nov 01 20:15:00.143433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143430    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab900363-9fbb-41fe-8e89-b27501cb7f41" containerName="revoke-keys"
Nov 01 20:15:00.143742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143447    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc243873-82c7-43d8-8851-1334401203b2" containerName="revoke-keys"
Nov 01 20:15:00.143742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143455    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc243873-82c7-43d8-8851-1334401203b2" containerName="revoke-keys"
Nov 01 20:15:00.143742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143495    5041 cpu_manager.go:414] "containerMap: removing container" podUID="83737883-64a0-4dcb-b495-6d5ffd6d0a1c" containerName="revoke-keys"
Nov 01 20:15:00.143742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143504    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83737883-64a0-4dcb-b495-6d5ffd6d0a1c" containerName="revoke-keys"
Nov 01 20:15:00.143742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143734    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab900363-9fbb-41fe-8e89-b27501cb7f41" containerName="revoke-keys"
Nov 01 20:15:00.144073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143755    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc243873-82c7-43d8-8851-1334401203b2" containerName="revoke-keys"
Nov 01 20:15:00.144073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.143772    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="83737883-64a0-4dcb-b495-6d5ffd6d0a1c" containerName="revoke-keys"
Nov 01 20:15:00.148243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.148225    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.148993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.148975    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:00.153189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.153168    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.153794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.153767    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:00.161109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.160802    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:00.161523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.161353    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.166044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.166022    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.167420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.167397    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:00.173086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.173070    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.173540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.173521    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:00.180122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.180058    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:00.186886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.186864    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:00.197834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.195797    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:00.198914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.198897    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:00.205478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.204771    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:00.211643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.211622    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.215402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.215383    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:00.245090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245054    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75cs\" (UniqueName: \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.245295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245108    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.245295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245149    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.245295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245206    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6bj\" (UniqueName: \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.245536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245321    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkkt\" (UniqueName: \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.245536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245403    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.245536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245491    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9fj\" (UniqueName: \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.245740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.245545    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.347145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347101    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9fj\" (UniqueName: \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.347356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347161    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.347356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347200    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zqs\" (UniqueName: \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.347356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347276    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75cs\" (UniqueName: \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.347356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347303    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.347356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347328    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.347641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347401    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.347641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347451    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6bj\" (UniqueName: \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.347641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347515    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.347641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347562    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkkt\" (UniqueName: \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.348118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347664    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.348118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.347706    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjj8\" (UniqueName: \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.348118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.348083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.348262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.348119    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.348262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.348152    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.348346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.348326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.355481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.355451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75cs\" (UniqueName: \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\") pod \"maas-api-key-cleanup-29892735-p2gq2\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.355606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.355577    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9fj\" (UniqueName: \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\") pod \"maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.355702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.355681    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6bj\" (UniqueName: \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.355743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.355702    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkkt\" (UniqueName: \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.449324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.449295    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjj8\" (UniqueName: \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.449465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.449369    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zqs\" (UniqueName: \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.449465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.449409    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.449594 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.449466    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.450307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.450286    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.450378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.450303    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.458211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.458187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zqs\" (UniqueName: \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\") pod \"maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.458307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.458259    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjj8\" (UniqueName: \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.478850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.478815    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:00.488149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.488126    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:00.499655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.499635    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:00.510698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.510673    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:00.520600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.519975    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:00.558994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.549871    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:00.588808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.588759    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh" event={"ID":"83737883-64a0-4dcb-b495-6d5ffd6d0a1c","Type":"ContainerDied","Data":"cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20"}
Nov 01 20:15:00.588976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.588839    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf6c7196f664971822c4d4dc6f2d28ca50d9cfb55014503dbe5215c66eaa3f20"
Nov 01 20:15:00.588976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.588935    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"
Nov 01 20:15:00.782611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.780611    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:00.799822 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:00.786975    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31cd78e_4178_441a_a08c_83a16aaae716.slice/crio-2e8ab1ffd9b65fcb8e7dadf3fa93e4107ecd32216e3ba5bc311f6ae0af2ea9bf WatchSource:0}: Error finding container 2e8ab1ffd9b65fcb8e7dadf3fa93e4107ecd32216e3ba5bc311f6ae0af2ea9bf: Status 404 returned error can't find the container with id 2e8ab1ffd9b65fcb8e7dadf3fa93e4107ecd32216e3ba5bc311f6ae0af2ea9bf
Nov 01 20:15:00.799822 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:00.787634    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb4ae74_ddeb_4d97_a52c_6e0b60bd220c.slice/crio-f7b1e79b07bae3bcf9685578bcf37cabfa4706ce985e2857c069b4294da0b8fb WatchSource:0}: Error finding container f7b1e79b07bae3bcf9685578bcf37cabfa4706ce985e2857c069b4294da0b8fb: Status 404 returned error can't find the container with id f7b1e79b07bae3bcf9685578bcf37cabfa4706ce985e2857c069b4294da0b8fb
Nov 01 20:15:00.799822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.788073    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:00.912509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.912437    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:00.912509 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:00.912450    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258c20bd_bb37_4dc3_84f2_d6b70f752d55.slice/crio-ccd2800a5afc2b03b6edb6a86fc7dfaaff9c6882a7a67c22a9b50624401a5f96 WatchSource:0}: Error finding container ccd2800a5afc2b03b6edb6a86fc7dfaaff9c6882a7a67c22a9b50624401a5f96: Status 404 returned error can't find the container with id ccd2800a5afc2b03b6edb6a86fc7dfaaff9c6882a7a67c22a9b50624401a5f96
Nov 01 20:15:00.919616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:00.919593    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:01.025692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.025600    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"]
Nov 01 20:15:01.029549 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:01.029517    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79302370_9894_423d_9d70_90e6deeed94f.slice/crio-1dc6e8e44b41f7d3ed0c46d09e0fab2d03ad95ab3b398e0aae4549c9316a34df WatchSource:0}: Error finding container 1dc6e8e44b41f7d3ed0c46d09e0fab2d03ad95ab3b398e0aae4549c9316a34df: Status 404 returned error can't find the container with id 1dc6e8e44b41f7d3ed0c46d09e0fab2d03ad95ab3b398e0aae4549c9316a34df
Nov 01 20:15:01.031316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.031295    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc32c16c-gbs9j"]
Nov 01 20:15:01.033523 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:01.033497    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39c10e6_df4c_4e9e_9395_ea16bd98ddee.slice/crio-faa386d5ee8fcbd9b886800ba33514c3e4c3fbd2b8d9fbf9e0c5ec68f07432b0 WatchSource:0}: Error finding container faa386d5ee8fcbd9b886800ba33514c3e4c3fbd2b8d9fbf9e0c5ec68f07432b0: Status 404 returned error can't find the container with id faa386d5ee8fcbd9b886800ba33514c3e4c3fbd2b8d9fbf9e0c5ec68f07432b0
Nov 01 20:15:01.036357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.036335    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:01.051094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.051075    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:01.616768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.616689    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerStarted","Data":"308c2e5d43e9a58c65f15a209d4a21ddc98f54afdb2875e91d6a21a51294cd43"}
Nov 01 20:15:01.616947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.616771    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerStarted","Data":"ccd2800a5afc2b03b6edb6a86fc7dfaaff9c6882a7a67c22a9b50624401a5f96"}
Nov 01 20:15:01.619505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.619468    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerStarted","Data":"956d8322d4c556ccbd8941bf5d9ad894823be9382f895d7422cbedfdcc7af349"}
Nov 01 20:15:01.619611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.619524    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerStarted","Data":"2e8ab1ffd9b65fcb8e7dadf3fa93e4107ecd32216e3ba5bc311f6ae0af2ea9bf"}
Nov 01 20:15:01.620927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.620901    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerStarted","Data":"faa386d5ee8fcbd9b886800ba33514c3e4c3fbd2b8d9fbf9e0c5ec68f07432b0"}
Nov 01 20:15:01.622642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.622616    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerStarted","Data":"c6e162b659b2fc3628d22e32fd17a3933cd2ea27b186d9f4b0d113f518bb7b88"}
Nov 01 20:15:01.622849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.622825    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerStarted","Data":"f7b1e79b07bae3bcf9685578bcf37cabfa4706ce985e2857c069b4294da0b8fb"}
Nov 01 20:15:01.624398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.624367    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerStarted","Data":"c53006a7e0042cbc91a2c034e9dd30319aa4e549adbc886061b1eb61126b5bdd"}
Nov 01 20:15:01.624504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.624421    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerStarted","Data":"cd93fefb049801a9cd6bd04be489ce3bb8ec089eb475a3558a866233b2c346e7"}
Nov 01 20:15:01.626033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.626012    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerStarted","Data":"1dc6e8e44b41f7d3ed0c46d09e0fab2d03ad95ab3b398e0aae4549c9316a34df"}
Nov 01 20:15:01.636479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.636433    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" podStartSLOduration=1.636420454 podStartE2EDuration="1.636420454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:01.632017386 +0000 UTC m=+989.353096064" watchObservedRunningTime="2026-11-01 20:15:01.636420454 +0000 UTC m=+989.357499151"
Nov 01 20:15:01.651340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.651292    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" podStartSLOduration=1.6512772230000001 podStartE2EDuration="1.651277223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:01.647401775 +0000 UTC m=+989.368480451" watchObservedRunningTime="2026-11-01 20:15:01.651277223 +0000 UTC m=+989.372355899"
Nov 01 20:15:01.668173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.668122    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" podStartSLOduration=1.66810434 podStartE2EDuration="1.66810434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:01.662063825 +0000 UTC m=+989.383142503" watchObservedRunningTime="2026-11-01 20:15:01.66810434 +0000 UTC m=+989.389183017"
Nov 01 20:15:01.681032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:01.680981    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" podStartSLOduration=1.680967689 podStartE2EDuration="1.680967689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:01.678647883 +0000 UTC m=+989.399726593" watchObservedRunningTime="2026-11-01 20:15:01.680967689 +0000 UTC m=+989.402046365"
Nov 01 20:15:02.028286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.028251    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"]
Nov 01 20:15:02.033519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.033490    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9c6264-bv9s8"]
Nov 01 20:15:02.431265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.431190    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f44b95-f366-40ef-9895-69b6ca1bbd7a" path="/var/lib/kubelet/pods/35f44b95-f366-40ef-9895-69b6ca1bbd7a/volumes"
Nov 01 20:15:02.431701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.431687    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb685d55-1973-4d42-be9e-6d6f0dd9bff8" path="/var/lib/kubelet/pods/bb685d55-1973-4d42-be9e-6d6f0dd9bff8/volumes"
Nov 01 20:15:02.637464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.637428    5041 generic.go:356] "Generic (PLEG): container finished" podID="c31cd78e-4178-441a-a08c-83a16aaae716" containerID="956d8322d4c556ccbd8941bf5d9ad894823be9382f895d7422cbedfdcc7af349" exitCode=60
Nov 01 20:15:02.637646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.637517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerDied","Data":"956d8322d4c556ccbd8941bf5d9ad894823be9382f895d7422cbedfdcc7af349"}
Nov 01 20:15:02.637852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.637836    5041 scope.go:122] "RemoveContainer" containerID="956d8322d4c556ccbd8941bf5d9ad894823be9382f895d7422cbedfdcc7af349"
Nov 01 20:15:02.639368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.639340    5041 generic.go:356] "Generic (PLEG): container finished" podID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerID="c086d6ea6431d51b19976156116bbdb77722f4d0c7a34c47967826e50ba8198a" exitCode=60
Nov 01 20:15:02.639480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.639439    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerDied","Data":"c086d6ea6431d51b19976156116bbdb77722f4d0c7a34c47967826e50ba8198a"}
Nov 01 20:15:02.639667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.639649    5041 scope.go:122] "RemoveContainer" containerID="c086d6ea6431d51b19976156116bbdb77722f4d0c7a34c47967826e50ba8198a"
Nov 01 20:15:02.641565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.641492    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerID="c6e162b659b2fc3628d22e32fd17a3933cd2ea27b186d9f4b0d113f518bb7b88" exitCode=60
Nov 01 20:15:02.641565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.641520    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerDied","Data":"c6e162b659b2fc3628d22e32fd17a3933cd2ea27b186d9f4b0d113f518bb7b88"}
Nov 01 20:15:02.641890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.641748    5041 scope.go:122] "RemoveContainer" containerID="c6e162b659b2fc3628d22e32fd17a3933cd2ea27b186d9f4b0d113f518bb7b88"
Nov 01 20:15:02.644316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.644291    5041 generic.go:356] "Generic (PLEG): container finished" podID="0605c95f-ac79-4405-aefe-679e54f07852" containerID="c53006a7e0042cbc91a2c034e9dd30319aa4e549adbc886061b1eb61126b5bdd" exitCode=60
Nov 01 20:15:02.644415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.644390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerDied","Data":"c53006a7e0042cbc91a2c034e9dd30319aa4e549adbc886061b1eb61126b5bdd"}
Nov 01 20:15:02.644593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.644576    5041 scope.go:122] "RemoveContainer" containerID="c53006a7e0042cbc91a2c034e9dd30319aa4e549adbc886061b1eb61126b5bdd"
Nov 01 20:15:02.646276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.646256    5041 generic.go:356] "Generic (PLEG): container finished" podID="79302370-9894-423d-9d70-90e6deeed94f" containerID="b2a5f7271f95cc3e94835f44bea5f4c9b90a634f1a8228347606a2a9ed4edf62" exitCode=60
Nov 01 20:15:02.646428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.646410    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerDied","Data":"b2a5f7271f95cc3e94835f44bea5f4c9b90a634f1a8228347606a2a9ed4edf62"}
Nov 01 20:15:02.646683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.646666    5041 scope.go:122] "RemoveContainer" containerID="b2a5f7271f95cc3e94835f44bea5f4c9b90a634f1a8228347606a2a9ed4edf62"
Nov 01 20:15:02.648376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.648352    5041 generic.go:356] "Generic (PLEG): container finished" podID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerID="308c2e5d43e9a58c65f15a209d4a21ddc98f54afdb2875e91d6a21a51294cd43" exitCode=60
Nov 01 20:15:02.648462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.648382    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerDied","Data":"308c2e5d43e9a58c65f15a209d4a21ddc98f54afdb2875e91d6a21a51294cd43"}
Nov 01 20:15:02.648659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:02.648642    5041 scope.go:122] "RemoveContainer" containerID="308c2e5d43e9a58c65f15a209d4a21ddc98f54afdb2875e91d6a21a51294cd43"
Nov 01 20:15:03.084800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.078037    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:15:03.160712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.149907    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.160712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.154537    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-29b57c-openshift-default-dockercfg-6fnqw\""
Nov 01 20:15:03.215767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.215733    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.242950    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243117    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243180    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243224    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243360    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckfh\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243672    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243720    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243825    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.250900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.243884    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.345604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.345560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346062    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.345997    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346108    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346292    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346335    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346368    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346402    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckfh\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.346445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.346419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.347285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.347260    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.347431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.347301    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.347763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.347730    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.349595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.349572    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.349874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.349856    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.355289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.355263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckfh\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.355289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.355275    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.512439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.512349    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:03.690848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.690814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerStarted","Data":"50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"}
Nov 01 20:15:03.694002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.693971    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerStarted","Data":"bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"}
Nov 01 20:15:03.696464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.696437    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerStarted","Data":"a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"}
Nov 01 20:15:03.700270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.700241    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerStarted","Data":"d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"}
Nov 01 20:15:03.702756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.702722    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerStarted","Data":"c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"}
Nov 01 20:15:03.705308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.705277    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerID="ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37" exitCode=60
Nov 01 20:15:03.705407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.705361    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerDied","Data":"ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37"}
Nov 01 20:15:03.705473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.705423    5041 scope.go:122] "RemoveContainer" containerID="c6e162b659b2fc3628d22e32fd17a3933cd2ea27b186d9f4b0d113f518bb7b88"
Nov 01 20:15:03.705716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.705688    5041 scope.go:122] "RemoveContainer" containerID="ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37"
Nov 01 20:15:03.707083 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:03.706622    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv_odh-ai-gateway-infra(1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c"
Nov 01 20:15:03.722601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.722559    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" podStartSLOduration=3.7225451019999998 podStartE2EDuration="3.722545102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:03.721495483 +0000 UTC m=+991.442574160" watchObservedRunningTime="2026-11-01 20:15:03.722545102 +0000 UTC m=+991.443623777"
Nov 01 20:15:03.789668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.789618    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" podStartSLOduration=3.789602373 podStartE2EDuration="3.789602373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:03.784375985 +0000 UTC m=+991.505454662" watchObservedRunningTime="2026-11-01 20:15:03.789602373 +0000 UTC m=+991.510681049"
Nov 01 20:15:03.806832 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:03.806767    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11cbcc_3bf9_445f_8a97_3f79957e62b6.slice/crio-d55fc3302948593f0bee6ce6ad7835badc0574d68c2e3ce38e21b6c22343ceff WatchSource:0}: Error finding container d55fc3302948593f0bee6ce6ad7835badc0574d68c2e3ce38e21b6c22343ceff: Status 404 returned error can't find the container with id d55fc3302948593f0bee6ce6ad7835badc0574d68c2e3ce38e21b6c22343ceff
Nov 01 20:15:03.808310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.808287    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:15:03.809514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.809477    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:15:03.809610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.809537    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:15:03.809610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:03.809572    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:15:04.396247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.395886    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"]
Nov 01 20:15:04.401438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.401410    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-912c8c21-w8jwb"]
Nov 01 20:15:04.432762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.432728    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab900363-9fbb-41fe-8e89-b27501cb7f41" path="/var/lib/kubelet/pods/ab900363-9fbb-41fe-8e89-b27501cb7f41/volumes"
Nov 01 20:15:04.723644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.723609    5041 generic.go:356] "Generic (PLEG): container finished" podID="79302370-9894-423d-9d70-90e6deeed94f" containerID="bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c" exitCode=60
Nov 01 20:15:04.723873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.723685    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerDied","Data":"bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"}
Nov 01 20:15:04.723873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.723737    5041 scope.go:122] "RemoveContainer" containerID="b2a5f7271f95cc3e94835f44bea5f4c9b90a634f1a8228347606a2a9ed4edf62"
Nov 01 20:15:04.724124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.724107    5041 scope.go:122] "RemoveContainer" containerID="bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"
Nov 01 20:15:04.724457 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.724436    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq_odh-ai-gateway-infra(79302370-9894-423d-9d70-90e6deeed94f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" podUID="79302370-9894-423d-9d70-90e6deeed94f"
Nov 01 20:15:04.725732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.725710    5041 generic.go:356] "Generic (PLEG): container finished" podID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerID="a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187" exitCode=60
Nov 01 20:15:04.725861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.725795    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerDied","Data":"a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"}
Nov 01 20:15:04.726065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.726051    5041 scope.go:122] "RemoveContainer" containerID="a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"
Nov 01 20:15:04.726360 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.726340    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t_odh-ai-gateway-infra(258c20bd-bb37-4dc3-84f2-d6b70f752d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55"
Nov 01 20:15:04.727765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.727746    5041 generic.go:356] "Generic (PLEG): container finished" podID="c31cd78e-4178-441a-a08c-83a16aaae716" containerID="d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe" exitCode=60
Nov 01 20:15:04.727880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.727835    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerDied","Data":"d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"}
Nov 01 20:15:04.728175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.728155    5041 scope.go:122] "RemoveContainer" containerID="d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"
Nov 01 20:15:04.728482 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.728464    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-p2gq2_odh-ai-gateway-infra(c31cd78e-4178-441a-a08c-83a16aaae716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" podUID="c31cd78e-4178-441a-a08c-83a16aaae716"
Nov 01 20:15:04.729942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.729925    5041 generic.go:356] "Generic (PLEG): container finished" podID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerID="c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b" exitCode=60
Nov 01 20:15:04.730011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.729994    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerDied","Data":"c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"}
Nov 01 20:15:04.730183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.730169    5041 scope.go:122] "RemoveContainer" containerID="c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"
Nov 01 20:15:04.730398 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.730376    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25_odh-ai-gateway-infra(c39c10e6-df4c-4e9e-9395-ea16bd98ddee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee"
Nov 01 20:15:04.731813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.731793    5041 scope.go:122] "RemoveContainer" containerID="ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37"
Nov 01 20:15:04.732026 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.732004    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv_odh-ai-gateway-infra(1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c"
Nov 01 20:15:04.733285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.733266    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" event={"ID":"fd11cbcc-3bf9-445f-8a97-3f79957e62b6","Type":"ContainerStarted","Data":"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"}
Nov 01 20:15:04.733388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.733303    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" event={"ID":"fd11cbcc-3bf9-445f-8a97-3f79957e62b6","Type":"ContainerStarted","Data":"d55fc3302948593f0bee6ce6ad7835badc0574d68c2e3ce38e21b6c22343ceff"}
Nov 01 20:15:04.735015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.734992    5041 generic.go:356] "Generic (PLEG): container finished" podID="0605c95f-ac79-4405-aefe-679e54f07852" containerID="50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004" exitCode=60
Nov 01 20:15:04.735102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.735049    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerDied","Data":"50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"}
Nov 01 20:15:04.735269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.735256    5041 scope.go:122] "RemoveContainer" containerID="50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"
Nov 01 20:15:04.735450 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:04.735435    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r_odh-ai-gateway-infra(0605c95f-ac79-4405-aefe-679e54f07852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" podUID="0605c95f-ac79-4405-aefe-679e54f07852"
Nov 01 20:15:04.739560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.739542    5041 scope.go:122] "RemoveContainer" containerID="308c2e5d43e9a58c65f15a209d4a21ddc98f54afdb2875e91d6a21a51294cd43"
Nov 01 20:15:04.793312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.793285    5041 scope.go:122] "RemoveContainer" containerID="956d8322d4c556ccbd8941bf5d9ad894823be9382f895d7422cbedfdcc7af349"
Nov 01 20:15:04.814487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.814460    5041 scope.go:122] "RemoveContainer" containerID="c086d6ea6431d51b19976156116bbdb77722f4d0c7a34c47967826e50ba8198a"
Nov 01 20:15:04.832662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.832636    5041 scope.go:122] "RemoveContainer" containerID="c53006a7e0042cbc91a2c034e9dd30319aa4e549adbc886061b1eb61126b5bdd"
Nov 01 20:15:04.899654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:04.899453    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podStartSLOduration=1.899430008 podStartE2EDuration="1.899430008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:04.876914553 +0000 UTC m=+992.597993276" watchObservedRunningTime="2026-11-01 20:15:04.899430008 +0000 UTC m=+992.620508685"
Nov 01 20:15:05.513415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.513385    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:05.515445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.515412    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 20:15:05.515539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.515468    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 20:15:05.773537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.773458    5041 scope.go:122] "RemoveContainer" containerID="50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"
Nov 01 20:15:05.773887 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:05.773855    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r_odh-ai-gateway-infra(0605c95f-ac79-4405-aefe-679e54f07852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" podUID="0605c95f-ac79-4405-aefe-679e54f07852"
Nov 01 20:15:05.775553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.775533    5041 scope.go:122] "RemoveContainer" containerID="bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"
Nov 01 20:15:05.775886 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:05.775855    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq_odh-ai-gateway-infra(79302370-9894-423d-9d70-90e6deeed94f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" podUID="79302370-9894-423d-9d70-90e6deeed94f"
Nov 01 20:15:05.777596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.777576    5041 scope.go:122] "RemoveContainer" containerID="a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"
Nov 01 20:15:05.777891 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:05.777873    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t_odh-ai-gateway-infra(258c20bd-bb37-4dc3-84f2-d6b70f752d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55"
Nov 01 20:15:05.779476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.779457    5041 scope.go:122] "RemoveContainer" containerID="d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"
Nov 01 20:15:05.779695 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:05.779673    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-p2gq2_odh-ai-gateway-infra(c31cd78e-4178-441a-a08c-83a16aaae716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" podUID="c31cd78e-4178-441a-a08c-83a16aaae716"
Nov 01 20:15:05.781040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:05.781022    5041 scope.go:122] "RemoveContainer" containerID="c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"
Nov 01 20:15:05.781244 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:05.781225    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25_odh-ai-gateway-infra(c39c10e6-df4c-4e9e-9395-ea16bd98ddee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee"
Nov 01 20:15:06.512861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:06.512825    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 20:15:06.513040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:06.512883    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 20:15:07.512915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:07.512878    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 20:15:07.513288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:07.512931    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 20:15:08.512913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:08.512873    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 20:15:08.513327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:08.512929    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 20:15:09.513715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:09.513679    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 01 20:15:09.514159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:09.513731    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 01 20:15:10.554304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:10.554265    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:10.555247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:10.555203    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:10.556091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:10.556062    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:15:13.807806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:13.793994    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:13.832678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:13.832635    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:14.261974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:14.261866    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:15.421105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:15.421077    5041 scope.go:122] "RemoveContainer" containerID="c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"
Nov 01 20:15:16.189455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:16.189411    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerStarted","Data":"2b4e08bb1659bd1cd32b09d424c666e2f445b8432c990b73a1838fa201964687"}
Nov 01 20:15:16.420981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:16.420945    5041 scope.go:122] "RemoveContainer" containerID="a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"
Nov 01 20:15:16.421143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:16.421020    5041 scope.go:122] "RemoveContainer" containerID="ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37"
Nov 01 20:15:16.421494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:16.421258    5041 scope.go:122] "RemoveContainer" containerID="d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"
Nov 01 20:15:17.200235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.200202    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerStarted","Data":"1ecec883551dab2e8d3a67f381e26ee889d741fbac7bcd0565bf02e00ccf88ba"}
Nov 01 20:15:17.202149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.202124    5041 generic.go:356] "Generic (PLEG): container finished" podID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerID="2b4e08bb1659bd1cd32b09d424c666e2f445b8432c990b73a1838fa201964687" exitCode=60
Nov 01 20:15:17.202283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.202192    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerDied","Data":"2b4e08bb1659bd1cd32b09d424c666e2f445b8432c990b73a1838fa201964687"}
Nov 01 20:15:17.202283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.202252    5041 scope.go:122] "RemoveContainer" containerID="c442fb9d2606656d5e61a275b38d7960ea57d4f2735af1905e2ca3bc329adc7b"
Nov 01 20:15:17.202533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.202519    5041 scope.go:122] "RemoveContainer" containerID="2b4e08bb1659bd1cd32b09d424c666e2f445b8432c990b73a1838fa201964687"
Nov 01 20:15:17.202835 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:17.202812    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25_odh-ai-gateway-infra(c39c10e6-df4c-4e9e-9395-ea16bd98ddee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee"
Nov 01 20:15:17.204448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.204427    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerStarted","Data":"2e61a0fe5702b39ca0cb0ecc6639e44ca44d6a98da354f143f270cb1907b1259"}
Nov 01 20:15:17.206285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.206267    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerStarted","Data":"7a403b3ddd6dcf8698dffc7d4ea6252a4aacaad77bfecbad508177145783febe"}
Nov 01 20:15:17.218158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.218136    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:17.420678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:17.420648    5041 scope.go:122] "RemoveContainer" containerID="50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"
Nov 01 20:15:18.216994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.216909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerStarted","Data":"6f5602b146e9609daab5b9f785bab426bfd33ce9009666b200d3efb1ac6086ec"}
Nov 01 20:15:18.219788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.219744    5041 generic.go:356] "Generic (PLEG): container finished" podID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerID="7a403b3ddd6dcf8698dffc7d4ea6252a4aacaad77bfecbad508177145783febe" exitCode=60
Nov 01 20:15:18.219901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.219810    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerDied","Data":"7a403b3ddd6dcf8698dffc7d4ea6252a4aacaad77bfecbad508177145783febe"}
Nov 01 20:15:18.219901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.219868    5041 scope.go:122] "RemoveContainer" containerID="a7086382131d995f4661e8f071768bb5d84e5f8d0e98f743690678ce8d613187"
Nov 01 20:15:18.220232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.220215    5041 scope.go:122] "RemoveContainer" containerID="7a403b3ddd6dcf8698dffc7d4ea6252a4aacaad77bfecbad508177145783febe"
Nov 01 20:15:18.220573 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:18.220550    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t_odh-ai-gateway-infra(258c20bd-bb37-4dc3-84f2-d6b70f752d55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55"
Nov 01 20:15:18.222050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.222030    5041 generic.go:356] "Generic (PLEG): container finished" podID="c31cd78e-4178-441a-a08c-83a16aaae716" containerID="1ecec883551dab2e8d3a67f381e26ee889d741fbac7bcd0565bf02e00ccf88ba" exitCode=60
Nov 01 20:15:18.222133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.222075    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerDied","Data":"1ecec883551dab2e8d3a67f381e26ee889d741fbac7bcd0565bf02e00ccf88ba"}
Nov 01 20:15:18.222395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.222382    5041 scope.go:122] "RemoveContainer" containerID="1ecec883551dab2e8d3a67f381e26ee889d741fbac7bcd0565bf02e00ccf88ba"
Nov 01 20:15:18.222627 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:18.222595    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892735-p2gq2_odh-ai-gateway-infra(c31cd78e-4178-441a-a08c-83a16aaae716)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" podUID="c31cd78e-4178-441a-a08c-83a16aaae716"
Nov 01 20:15:18.226174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.226151    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerID="2e61a0fe5702b39ca0cb0ecc6639e44ca44d6a98da354f143f270cb1907b1259" exitCode=60
Nov 01 20:15:18.226285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.226186    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerDied","Data":"2e61a0fe5702b39ca0cb0ecc6639e44ca44d6a98da354f143f270cb1907b1259"}
Nov 01 20:15:18.226497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.226480    5041 scope.go:122] "RemoveContainer" containerID="2e61a0fe5702b39ca0cb0ecc6639e44ca44d6a98da354f143f270cb1907b1259"
Nov 01 20:15:18.226733 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:18.226714    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv_odh-ai-gateway-infra(1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c"
Nov 01 20:15:18.237611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.237586    5041 scope.go:122] "RemoveContainer" containerID="d21617d43b556fa1f27b51e35f7be128fbb96b7e2469f78ef0824745e61e17fe"
Nov 01 20:15:18.238646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.238625    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:18.255634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.255604    5041 scope.go:122] "RemoveContainer" containerID="ddf8476418a2c9070ba08df3a2de0966eb19b097c3046518b42621fa15e6fe37"
Nov 01 20:15:18.258426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.258403    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:18.292929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.292892    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:18.424033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.422381    5041 scope.go:122] "RemoveContainer" containerID="bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"
Nov 01 20:15:18.450479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.450084    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:18.549331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.548765    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\" (UniqueName: \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\") pod \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") "
Nov 01 20:15:18.549331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.548950    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\") pod \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\" (UID: \"c39c10e6-df4c-4e9e-9395-ea16bd98ddee\") "
Nov 01 20:15:18.549567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.549542    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca" pod "c39c10e6-df4c-4e9e-9395-ea16bd98ddee" (UID: "c39c10e6-df4c-4e9e-9395-ea16bd98ddee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:18.549821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.549803    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:18.558405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.558376    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj" pod "c39c10e6-df4c-4e9e-9395-ea16bd98ddee" (UID: "c39c10e6-df4c-4e9e-9395-ea16bd98ddee"). InnerVolumeSpecName "kube-api-access-2g9fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:18.650867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:18.650825    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g9fj\" (UniqueName: \"kubernetes.io/projected/c39c10e6-df4c-4e9e-9395-ea16bd98ddee-kube-api-access-2g9fj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.025415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.025373    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"]
Nov 01 20:15:19.030649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.030609    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da89eacc-ntm6l"]
Nov 01 20:15:19.240672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.240586    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerStarted","Data":"e560adf9f9d6c94b4e949dd97cd6f9e746079798082fb19d439891077df86792"}
Nov 01 20:15:19.246584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.246552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25" event={"ID":"c39c10e6-df4c-4e9e-9395-ea16bd98ddee","Type":"ContainerDied","Data":"faa386d5ee8fcbd9b886800ba33514c3e4c3fbd2b8d9fbf9e0c5ec68f07432b0"}
Nov 01 20:15:19.246684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.246619    5041 scope.go:122] "RemoveContainer" containerID="2b4e08bb1659bd1cd32b09d424c666e2f445b8432c990b73a1838fa201964687"
Nov 01 20:15:19.246938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.246553    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"
Nov 01 20:15:19.252247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.252216    5041 generic.go:356] "Generic (PLEG): container finished" podID="0605c95f-ac79-4405-aefe-679e54f07852" containerID="6f5602b146e9609daab5b9f785bab426bfd33ce9009666b200d3efb1ac6086ec" exitCode=60
Nov 01 20:15:19.252355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.252294    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerDied","Data":"6f5602b146e9609daab5b9f785bab426bfd33ce9009666b200d3efb1ac6086ec"}
Nov 01 20:15:19.252725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.252705    5041 scope.go:122] "RemoveContainer" containerID="6f5602b146e9609daab5b9f785bab426bfd33ce9009666b200d3efb1ac6086ec"
Nov 01 20:15:19.253418 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:19.253035    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r_odh-ai-gateway-infra(0605c95f-ac79-4405-aefe-679e54f07852)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" podUID="0605c95f-ac79-4405-aefe-679e54f07852"
Nov 01 20:15:19.260369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.260341    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:19.265713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.263930    5041 scope.go:122] "RemoveContainer" containerID="50b24de314efbbf890a06ec47e024f5f1405a245e6a155622509282cf458f004"
Nov 01 20:15:19.364147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.364123    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:19.369111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.369090    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5370bb-29892735-vcx25"]
Nov 01 20:15:19.494588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.494521    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:19.553936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.553903    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:19.595165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.595126    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:19.596423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.596394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\") pod \"c31cd78e-4178-441a-a08c-83a16aaae716\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") "
Nov 01 20:15:19.596576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.596560    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\" (UniqueName: \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\") pod \"c31cd78e-4178-441a-a08c-83a16aaae716\" (UID: \"c31cd78e-4178-441a-a08c-83a16aaae716\") "
Nov 01 20:15:19.597362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.597334    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca" pod "c31cd78e-4178-441a-a08c-83a16aaae716" (UID: "c31cd78e-4178-441a-a08c-83a16aaae716"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:19.599455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.599428    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs" pod "c31cd78e-4178-441a-a08c-83a16aaae716" (UID: "c31cd78e-4178-441a-a08c-83a16aaae716"). InnerVolumeSpecName "kube-api-access-q75cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:19.697643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.697605    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\") pod \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") "
Nov 01 20:15:19.697876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.697715    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\" (UniqueName: \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\") pod \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\" (UID: \"258c20bd-bb37-4dc3-84f2-d6b70f752d55\") "
Nov 01 20:15:19.697876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.697864    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\") pod \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") "
Nov 01 20:15:19.698003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.697935    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\" (UniqueName: \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\") pod \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\" (UID: \"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c\") "
Nov 01 20:15:19.698140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.698107    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca" pod "258c20bd-bb37-4dc3-84f2-d6b70f752d55" (UID: "258c20bd-bb37-4dc3-84f2-d6b70f752d55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:19.698440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.698420    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31cd78e-4178-441a-a08c-83a16aaae716-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.698529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.698445    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75cs\" (UniqueName: \"kubernetes.io/projected/c31cd78e-4178-441a-a08c-83a16aaae716-kube-api-access-q75cs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.698529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.698418    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca" pod "1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" (UID: "1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:19.705827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.705421    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs" pod "258c20bd-bb37-4dc3-84f2-d6b70f752d55" (UID: "258c20bd-bb37-4dc3-84f2-d6b70f752d55"). InnerVolumeSpecName "kube-api-access-82zqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:19.706869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.706721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj" pod "1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" (UID: "1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c"). InnerVolumeSpecName "kube-api-access-jw6bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:19.799424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.799390    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/258c20bd-bb37-4dc3-84f2-d6b70f752d55-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.799424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.799418    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zqs\" (UniqueName: \"kubernetes.io/projected/258c20bd-bb37-4dc3-84f2-d6b70f752d55-kube-api-access-82zqs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.799424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.799428    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:19.799659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:19.799437    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6bj\" (UniqueName: \"kubernetes.io/projected/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c-kube-api-access-jw6bj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.265103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.264971    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"
Nov 01 20:15:20.265103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.264986    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t" event={"ID":"258c20bd-bb37-4dc3-84f2-d6b70f752d55","Type":"ContainerDied","Data":"ccd2800a5afc2b03b6edb6a86fc7dfaaff9c6882a7a67c22a9b50624401a5f96"}
Nov 01 20:15:20.265103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.265086    5041 scope.go:122] "RemoveContainer" containerID="7a403b3ddd6dcf8698dffc7d4ea6252a4aacaad77bfecbad508177145783febe"
Nov 01 20:15:20.268652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.268612    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"
Nov 01 20:15:20.268789 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.268649    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2" event={"ID":"c31cd78e-4178-441a-a08c-83a16aaae716","Type":"ContainerDied","Data":"2e8ab1ffd9b65fcb8e7dadf3fa93e4107ecd32216e3ba5bc311f6ae0af2ea9bf"}
Nov 01 20:15:20.273214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.273185    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:20.275528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.275510    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"
Nov 01 20:15:20.275899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.275667    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv" event={"ID":"1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c","Type":"ContainerDied","Data":"f7b1e79b07bae3bcf9685578bcf37cabfa4706ce985e2857c069b4294da0b8fb"}
Nov 01 20:15:20.281148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.281082    5041 generic.go:356] "Generic (PLEG): container finished" podID="79302370-9894-423d-9d70-90e6deeed94f" containerID="e560adf9f9d6c94b4e949dd97cd6f9e746079798082fb19d439891077df86792" exitCode=60
Nov 01 20:15:20.281262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.281171    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerDied","Data":"e560adf9f9d6c94b4e949dd97cd6f9e746079798082fb19d439891077df86792"}
Nov 01 20:15:20.287912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.287887    5041 scope.go:122] "RemoveContainer" containerID="1ecec883551dab2e8d3a67f381e26ee889d741fbac7bcd0565bf02e00ccf88ba"
Nov 01 20:15:20.310460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.310434    5041 scope.go:122] "RemoveContainer" containerID="2e61a0fe5702b39ca0cb0ecc6639e44ca44d6a98da354f143f270cb1907b1259"
Nov 01 20:15:20.337934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.337895    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:20.340887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.340812    5041 scope.go:122] "RemoveContainer" containerID="bb964f8992ca3cb6de246c6659125c5480a5f2cbbac1f225d31b52f26124aa0c"
Nov 01 20:15:20.346972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.346696    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e27ae7-29892735-kf65t"]
Nov 01 20:15:20.360644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.360612    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:20.368871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.368842    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892735-p2gq2"]
Nov 01 20:15:20.375585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.375540    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:20.387730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.387693    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892735-swbpv"]
Nov 01 20:15:20.436306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.436271    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" path="/var/lib/kubelet/pods/1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c/volumes"
Nov 01 20:15:20.437183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.437159    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" path="/var/lib/kubelet/pods/258c20bd-bb37-4dc3-84f2-d6b70f752d55/volumes"
Nov 01 20:15:20.437934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.437917    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31cdef5-7629-4ee2-8167-38ebb54362be" path="/var/lib/kubelet/pods/b31cdef5-7629-4ee2-8167-38ebb54362be/volumes"
Nov 01 20:15:20.439019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.439000    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" path="/var/lib/kubelet/pods/c31cd78e-4178-441a-a08c-83a16aaae716/volumes"
Nov 01 20:15:20.439680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.439664    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" path="/var/lib/kubelet/pods/c39c10e6-df4c-4e9e-9395-ea16bd98ddee/volumes"
Nov 01 20:15:20.517082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.517012    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:20.523085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.523059    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:20.613162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.613127    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\") pod \"79302370-9894-423d-9d70-90e6deeed94f\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") "
Nov 01 20:15:20.613373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.613362    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\" (UniqueName: \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\") pod \"0605c95f-ac79-4405-aefe-679e54f07852\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") "
Nov 01 20:15:20.613468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.613395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\") pod \"0605c95f-ac79-4405-aefe-679e54f07852\" (UID: \"0605c95f-ac79-4405-aefe-679e54f07852\") "
Nov 01 20:15:20.613468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.613432    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\" (UniqueName: \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\") pod \"79302370-9894-423d-9d70-90e6deeed94f\" (UID: \"79302370-9894-423d-9d70-90e6deeed94f\") "
Nov 01 20:15:20.614234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.614203    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca" pod "79302370-9894-423d-9d70-90e6deeed94f" (UID: "79302370-9894-423d-9d70-90e6deeed94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:20.614234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.614214    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca" pod "0605c95f-ac79-4405-aefe-679e54f07852" (UID: "0605c95f-ac79-4405-aefe-679e54f07852"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:20.616174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.616149    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt" pod "79302370-9894-423d-9d70-90e6deeed94f" (UID: "79302370-9894-423d-9d70-90e6deeed94f"). InnerVolumeSpecName "kube-api-access-pxkkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:20.616174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.616163    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8" pod "0605c95f-ac79-4405-aefe-679e54f07852" (UID: "0605c95f-ac79-4405-aefe-679e54f07852"). InnerVolumeSpecName "kube-api-access-qrjj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:20.715387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.715342    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjj8\" (UniqueName: \"kubernetes.io/projected/0605c95f-ac79-4405-aefe-679e54f07852-kube-api-access-qrjj8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.715387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.715385    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0605c95f-ac79-4405-aefe-679e54f07852-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.715635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.715400    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkkt\" (UniqueName: \"kubernetes.io/projected/79302370-9894-423d-9d70-90e6deeed94f-kube-api-access-pxkkt\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:20.715635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:20.715413    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79302370-9894-423d-9d70-90e6deeed94f-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:21.291518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.291490    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"
Nov 01 20:15:21.291960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.291531    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq" event={"ID":"79302370-9894-423d-9d70-90e6deeed94f","Type":"ContainerDied","Data":"1dc6e8e44b41f7d3ed0c46d09e0fab2d03ad95ab3b398e0aae4549c9316a34df"}
Nov 01 20:15:21.291960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.291590    5041 scope.go:122] "RemoveContainer" containerID="e560adf9f9d6c94b4e949dd97cd6f9e746079798082fb19d439891077df86792"
Nov 01 20:15:21.299077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.299056    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"
Nov 01 20:15:21.299077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.299059    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r" event={"ID":"0605c95f-ac79-4405-aefe-679e54f07852","Type":"ContainerDied","Data":"cd93fefb049801a9cd6bd04be489ce3bb8ec089eb475a3558a866233b2c346e7"}
Nov 01 20:15:21.304402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.304384    5041 scope.go:122] "RemoveContainer" containerID="6f5602b146e9609daab5b9f785bab426bfd33ce9009666b200d3efb1ac6086ec"
Nov 01 20:15:21.318135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.318110    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:21.324223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.324198    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892735-b68kq"]
Nov 01 20:15:21.336060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.336037    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:21.341765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:21.341742    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892735-gfx8r"]
Nov 01 20:15:22.429731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:22.429699    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0605c95f-ac79-4405-aefe-679e54f07852" path="/var/lib/kubelet/pods/0605c95f-ac79-4405-aefe-679e54f07852/volumes"
Nov 01 20:15:22.430243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:22.430228    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79302370-9894-423d-9d70-90e6deeed94f" path="/var/lib/kubelet/pods/79302370-9894-423d-9d70-90e6deeed94f/volumes"
Nov 01 20:15:26.045801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:26.043923    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"]
Nov 01 20:15:26.056814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:26.052563    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601ecc0a-f9rz6"]
Nov 01 20:15:26.477164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:26.475225    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3981c54-c9bf-4d74-a9c1-f77c98c81a07" path="/var/lib/kubelet/pods/a3981c54-c9bf-4d74-a9c1-f77c98c81a07/volumes"
Nov 01 20:15:28.765803 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:28.765090    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:15:30.032801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:30.031894    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:30.264803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:30.263873    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:15:31.890387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.890344    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"]
Nov 01 20:15:31.891016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.890998    5041 cpu_manager.go:414] "containerMap: removing container" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.891016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891015    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891025    5041 cpu_manager.go:414] "containerMap: removing container" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891031    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891037    5041 cpu_manager.go:414] "containerMap: removing container" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891042    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891052    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891057    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891070    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891075    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891086    5041 cpu_manager.go:414] "containerMap: removing container" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891090    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891096    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891100    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891105    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891111    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891116    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891120    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891125    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891129    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891150    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.891150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891154    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891164    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891168    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891173    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891177    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891267    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891276    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891282    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891288    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891294    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891299    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891305    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891310    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891318    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891324    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891330    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31cd78e-4178-441a-a08c-83a16aaae716" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891335    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891339    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891347    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:31.892234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.891352    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:31.897191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.897166    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:31.902415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.902365    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:31.902415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.902404    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:15:31.902661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.902644    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:15:31.902755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.902712    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:15:31.902992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.902968    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"]
Nov 01 20:15:31.966021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.965985    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kmm\" (UniqueName: \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:31.966200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.966103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:31.966339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:31.966323    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.067584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.067551    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.067805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.067605    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kmm\" (UniqueName: \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.067805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.067651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.068313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.068289    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.069992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.069970    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.075604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.075577    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kmm\" (UniqueName: \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\") pod \"maas-api-revoke-sub-66f7552d-2djwt\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.216289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.216207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:32.506687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.506557    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"]
Nov 01 20:15:32.652404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:32.652366    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt" event={"ID":"271d2f68-4d40-42b4-8b6a-6dabc3fe3401","Type":"ContainerStarted","Data":"5f26a7d52831bc1837d4b0cb9e5135819ad2245f0029c5773f05d60daeb94d88"}
Nov 01 20:15:33.023941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.023851    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"]
Nov 01 20:15:33.029234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.029200    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50c82734-kwdvq"]
Nov 01 20:15:33.569134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.569108    5041 scope.go:122] "RemoveContainer" containerID="68760da68c26fe1fdad15559be409efe13a7b0d71342e67bf87ba616ba0331df"
Nov 01 20:15:33.616812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.615869    5041 scope.go:122] "RemoveContainer" containerID="067aefd68c69c3a10d7157a6d902da3c59bb7faa977c1001d7eeeabfed6c9ba6"
Nov 01 20:15:33.626531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.626507    5041 scope.go:122] "RemoveContainer" containerID="c36f78a7f350ba3b3e6f750996a01db99534c29a3553300205f75a61fb4deefc"
Nov 01 20:15:33.635822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.635803    5041 scope.go:122] "RemoveContainer" containerID="c13094b8c682854f8602b4492d1e3534fc5d4e5934c119f0318e24ae507aaded"
Nov 01 20:15:33.645116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.645096    5041 scope.go:122] "RemoveContainer" containerID="a7d99a56a853152ee965b5f4082093ba888ee6bd223dfdea7717152947449df6"
Nov 01 20:15:33.655107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.655085    5041 scope.go:122] "RemoveContainer" containerID="a7e6546c7b7e39051ca068c8e6667155fb14245944bc0a170f263826fc95e32d"
Nov 01 20:15:33.665713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.665679    5041 scope.go:122] "RemoveContainer" containerID="76ae381f976282ce1078ab2f3505b6a350cda2d88b17e0d70c1539c8745301c2"
Nov 01 20:15:33.668541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.668516    5041 generic.go:356] "Generic (PLEG): container finished" podID="271d2f68-4d40-42b4-8b6a-6dabc3fe3401" containerID="6eb2f28711c55f36c8384d6849f492f0dee145a9e59d414fe7366b9c6b382fa0" exitCode=0
Nov 01 20:15:33.668634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.668607    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt" event={"ID":"271d2f68-4d40-42b4-8b6a-6dabc3fe3401","Type":"ContainerDied","Data":"6eb2f28711c55f36c8384d6849f492f0dee145a9e59d414fe7366b9c6b382fa0"}
Nov 01 20:15:33.675578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.675558    5041 scope.go:122] "RemoveContainer" containerID="73283e3555ad759b4c33c8556570cc9d44112a2c77a98293c252f41ae13258ce"
Nov 01 20:15:33.688446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.688423    5041 scope.go:122] "RemoveContainer" containerID="ce2ec72747f978c03689459d2d67f7fc097bfe8c50330352e61df30614744320"
Nov 01 20:15:33.700182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.700161    5041 scope.go:122] "RemoveContainer" containerID="6d863433694a2b79f535e65817d5fbd518b6d90954245d5bffc888a9d971cf1b"
Nov 01 20:15:33.709891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.709867    5041 scope.go:122] "RemoveContainer" containerID="90a07340d3098a0c7e0c0698e6d9933f02ef63c7cf07c401de31064176d3cd5c"
Nov 01 20:15:33.722823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.722770    5041 scope.go:122] "RemoveContainer" containerID="ae5926f5457aaaa58bd9dc00aebbf76823903b0044e3d90b2a50b8c33eec582c"
Nov 01 20:15:33.732922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:33.732901    5041 scope.go:122] "RemoveContainer" containerID="1b200883ac34cd5b79bfd908017642e2186107c7ab7192a8ca95d16aff9ddda7"
Nov 01 20:15:34.430362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.429686    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b819a3c-1b13-486a-8784-e7472f9ab3b3" path="/var/lib/kubelet/pods/3b819a3c-1b13-486a-8784-e7472f9ab3b3/volumes"
Nov 01 20:15:34.858686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.858653    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:34.914859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.914824    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\") pod \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") "
Nov 01 20:15:34.915133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.915058    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\" (UniqueName: \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\") pod \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") "
Nov 01 20:15:34.915133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.915112    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\") pod \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\" (UID: \"271d2f68-4d40-42b4-8b6a-6dabc3fe3401\") "
Nov 01 20:15:34.915551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.915526    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca" pod "271d2f68-4d40-42b4-8b6a-6dabc3fe3401" (UID: "271d2f68-4d40-42b4-8b6a-6dabc3fe3401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:34.915734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.915713    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:34.917201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.917176    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca" pod "271d2f68-4d40-42b4-8b6a-6dabc3fe3401" (UID: "271d2f68-4d40-42b4-8b6a-6dabc3fe3401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:34.917306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:34.917250    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm" pod "271d2f68-4d40-42b4-8b6a-6dabc3fe3401" (UID: "271d2f68-4d40-42b4-8b6a-6dabc3fe3401"). InnerVolumeSpecName "kube-api-access-z8kmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:35.017377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:35.017339    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kmm\" (UniqueName: \"kubernetes.io/projected/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-kube-api-access-z8kmm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:35.017377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:35.017372    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271d2f68-4d40-42b4-8b6a-6dabc3fe3401-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:35.692458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:35.692419    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt" event={"ID":"271d2f68-4d40-42b4-8b6a-6dabc3fe3401","Type":"ContainerDied","Data":"5f26a7d52831bc1837d4b0cb9e5135819ad2245f0029c5773f05d60daeb94d88"}
Nov 01 20:15:35.692458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:35.692438    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"
Nov 01 20:15:35.692901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:35.692472    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f26a7d52831bc1837d4b0cb9e5135819ad2245f0029c5773f05d60daeb94d88"
Nov 01 20:15:36.549106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549076    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"]
Nov 01 20:15:36.549697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549684    5041 cpu_manager.go:414] "containerMap: removing container" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:36.549744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549698    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:36.549744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549718    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:36.549744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549724    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:36.549744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549738    5041 cpu_manager.go:414] "containerMap: removing container" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:36.549744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549744    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258c20bd-bb37-4dc3-84f2-d6b70f752d55" containerName="cleanup"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549751    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549756    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549761    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549766    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0605c95f-ac79-4405-aefe-679e54f07852" containerName="cleanup"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549808    5041 cpu_manager.go:414] "containerMap: removing container" podUID="271d2f68-4d40-42b4-8b6a-6dabc3fe3401" containerName="revoke-keys"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549816    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271d2f68-4d40-42b4-8b6a-6dabc3fe3401" containerName="revoke-keys"
Nov 01 20:15:36.549918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549917    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39c10e6-df4c-4e9e-9395-ea16bd98ddee" containerName="cleanup"
Nov 01 20:15:36.550127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549932    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="79302370-9894-423d-9d70-90e6deeed94f" containerName="cleanup"
Nov 01 20:15:36.550127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549940    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4ae74-ddeb-4d97-a52c-6e0b60bd220c" containerName="cleanup"
Nov 01 20:15:36.550127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.549949    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="271d2f68-4d40-42b4-8b6a-6dabc3fe3401" containerName="revoke-keys"
Nov 01 20:15:36.553332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.553312    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.557719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.556999    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:15:36.557719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.557086    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:15:36.557719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.557140    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:36.557719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.557231    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:15:36.571824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.562138    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"]
Nov 01 20:15:36.639855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.639804    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bpbm\" (UniqueName: \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.640032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.640013    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.640093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.640078    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.741985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.741951    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bpbm\" (UniqueName: \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.742482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.742057    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.742482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.742095    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.742895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.742874    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.744466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.744441    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.749932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.749912    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bpbm\" (UniqueName: \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\") pod \"maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:36.873168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:36.873093    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:37.379862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.379824    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"]
Nov 01 20:15:37.389256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.389218    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"]
Nov 01 20:15:37.389415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.389341    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.393250 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:37.392818    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf916d1d5_a0dd_469f_aa88_a30f609ccec9.slice/crio-0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3 WatchSource:0}: Error finding container 0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3: Status 404 returned error can't find the container with id 0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3
Nov 01 20:15:37.399241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.399215    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"]
Nov 01 20:15:37.463934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.463710    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxbj\" (UniqueName: \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.463934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.463839    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.463934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.463899    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.565072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.565029    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxbj\" (UniqueName: \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.565258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.565241    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.565422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.565397    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.566130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.566100    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.567821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.567799    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.572818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.572798    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxbj\" (UniqueName: \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\") pod \"maas-api-revoke-sub-84612ea4-xlhcc\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:37.713266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.713175    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj" event={"ID":"f916d1d5-a0dd-469f-aa88-a30f609ccec9","Type":"ContainerStarted","Data":"0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3"}
Nov 01 20:15:37.740059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:37.740026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:38.024743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.024721    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"]
Nov 01 20:15:38.027685 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:38.027656    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1b925d_4f17_4a37_bf8a_ea3cc8e5a6f8.slice/crio-a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e WatchSource:0}: Error finding container a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e: Status 404 returned error can't find the container with id a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e
Nov 01 20:15:38.030835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.030773    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"]
Nov 01 20:15:38.036345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.036321    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f2b979-dt6rz"]
Nov 01 20:15:38.429474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.429431    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d44a85-a3bf-4820-8208-d9e50665b218" path="/var/lib/kubelet/pods/80d44a85-a3bf-4820-8208-d9e50665b218/volumes"
Nov 01 20:15:38.724755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.724665    5041 generic.go:356] "Generic (PLEG): container finished" podID="f916d1d5-a0dd-469f-aa88-a30f609ccec9" containerID="a51e44640973c70ea6442df87dc54add4ae9aaf860ec78430a7b9fb9f78f9ac7" exitCode=0
Nov 01 20:15:38.724904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.724758    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj" event={"ID":"f916d1d5-a0dd-469f-aa88-a30f609ccec9","Type":"ContainerDied","Data":"a51e44640973c70ea6442df87dc54add4ae9aaf860ec78430a7b9fb9f78f9ac7"}
Nov 01 20:15:38.726626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.726604    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc" event={"ID":"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8","Type":"ContainerStarted","Data":"eb080219d2a6c2bc76b63de0295cb9672dfce825f9fd8cb53227dfc01f7f4ac9"}
Nov 01 20:15:38.726727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.726644    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc" event={"ID":"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8","Type":"ContainerStarted","Data":"a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e"}
Nov 01 20:15:38.755263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:38.755213    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc" podStartSLOduration=1.755199167 podStartE2EDuration="1.755199167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:38.752946283 +0000 UTC m=+1026.474024962" watchObservedRunningTime="2026-11-01 20:15:38.755199167 +0000 UTC m=+1026.476277869"
Nov 01 20:15:39.745223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:39.745185    5041 generic.go:356] "Generic (PLEG): container finished" podID="fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" containerID="eb080219d2a6c2bc76b63de0295cb9672dfce825f9fd8cb53227dfc01f7f4ac9" exitCode=0
Nov 01 20:15:39.745717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:39.745287    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc" event={"ID":"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8","Type":"ContainerDied","Data":"eb080219d2a6c2bc76b63de0295cb9672dfce825f9fd8cb53227dfc01f7f4ac9"}
Nov 01 20:15:39.954336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:39.954310    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:40.103815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.103766    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\" (UniqueName: \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\") pod \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") "
Nov 01 20:15:40.103981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.103966    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\") pod \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") "
Nov 01 20:15:40.104262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.104242    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\") pod \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\" (UID: \"f916d1d5-a0dd-469f-aa88-a30f609ccec9\") "
Nov 01 20:15:40.104375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.104331    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca" pod "f916d1d5-a0dd-469f-aa88-a30f609ccec9" (UID: "f916d1d5-a0dd-469f-aa88-a30f609ccec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:40.104699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.104682    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:40.106057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.106032    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm" pod "f916d1d5-a0dd-469f-aa88-a30f609ccec9" (UID: "f916d1d5-a0dd-469f-aa88-a30f609ccec9"). InnerVolumeSpecName "kube-api-access-5bpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:40.106292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.106260    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca" pod "f916d1d5-a0dd-469f-aa88-a30f609ccec9" (UID: "f916d1d5-a0dd-469f-aa88-a30f609ccec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:40.206333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.206296    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f916d1d5-a0dd-469f-aa88-a30f609ccec9-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:40.206333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.206330    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bpbm\" (UniqueName: \"kubernetes.io/projected/f916d1d5-a0dd-469f-aa88-a30f609ccec9-kube-api-access-5bpbm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:40.756943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.756854    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"
Nov 01 20:15:40.756943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.756882    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj" event={"ID":"f916d1d5-a0dd-469f-aa88-a30f609ccec9","Type":"ContainerDied","Data":"0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3"}
Nov 01 20:15:40.756943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.756941    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a263fdda7dbcf8c575d9181e468471e801fbd760fe430597f6e014ca376c7e3"
Nov 01 20:15:40.962219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:40.962190    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:41.119513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.119481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\") pod \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") "
Nov 01 20:15:41.119699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.119680    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\") pod \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") "
Nov 01 20:15:41.119749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.119721    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\" (UniqueName: \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\") pod \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\" (UID: \"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8\") "
Nov 01 20:15:41.119919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.119899    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca" pod "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" (UID: "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:41.120251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.120223    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:41.122064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.122033    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca" pod "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" (UID: "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:41.122410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.122385    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj" pod "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" (UID: "fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8"). InnerVolumeSpecName "kube-api-access-nvxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:41.221342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.221310    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:41.221342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.221338    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvxbj\" (UniqueName: \"kubernetes.io/projected/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8-kube-api-access-nvxbj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:41.767251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.767221    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"
Nov 01 20:15:41.767251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.767250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc" event={"ID":"fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8","Type":"ContainerDied","Data":"a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e"}
Nov 01 20:15:41.767811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:41.767291    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4ff55521fc326bc051bbd3f02f124ba488d8ce0ade7b7e76272777489b99b1e"
Nov 01 20:15:43.031011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.028153    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"]
Nov 01 20:15:43.062798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.048979    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ba8aa-tvjgs"]
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.395828    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"]
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397035    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f916d1d5-a0dd-469f-aa88-a30f609ccec9" containerName="revoke-keys"
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397057    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f916d1d5-a0dd-469f-aa88-a30f609ccec9" containerName="revoke-keys"
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397136    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" containerName="revoke-keys"
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397146    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" containerName="revoke-keys"
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397304    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f916d1d5-a0dd-469f-aa88-a30f609ccec9" containerName="revoke-keys"
Nov 01 20:15:43.400800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.397325    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" containerName="revoke-keys"
Nov 01 20:15:43.419239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.419215    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.425185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.425152    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:15:43.425463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.425440    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:43.425923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.425905    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:15:43.426071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.425971    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:15:43.428912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.428894    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"]
Nov 01 20:15:43.493825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.493757    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7pn8\" (UniqueName: \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.493997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.493954    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.494058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.493998    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.595479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.595441    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.595479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.595482    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.595700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.595660    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7pn8\" (UniqueName: \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.596162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.596140    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.598062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.598039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.604617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.604589    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7pn8\" (UniqueName: \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\") pod \"maas-api-revoke-sub-a162ac60-bk5nz\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:43.742975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:43.742890    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:44.059549 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:44.059519    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"]
Nov 01 20:15:44.060178 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:44.060153    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf721fd_7f50_4aab_97a3_c9f756c9728d.slice/crio-919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660 WatchSource:0}: Error finding container 919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660: Status 404 returned error can't find the container with id 919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660
Nov 01 20:15:44.429577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:44.429544    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a7cbca-51b4-48a5-9d4e-1991b98068ad" path="/var/lib/kubelet/pods/a1a7cbca-51b4-48a5-9d4e-1991b98068ad/volumes"
Nov 01 20:15:44.799415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:44.799381    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz" event={"ID":"1cf721fd-7f50-4aab-97a3-c9f756c9728d","Type":"ContainerStarted","Data":"91a84058d339bdb746042a8199d4ac67983530a99b5a92764311b401b42a9ede"}
Nov 01 20:15:44.799631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:44.799444    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz" event={"ID":"1cf721fd-7f50-4aab-97a3-c9f756c9728d","Type":"ContainerStarted","Data":"919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660"}
Nov 01 20:15:44.818662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:44.818608    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz" podStartSLOduration=1.818590956 podStartE2EDuration="1.818590956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:44.814109177 +0000 UTC m=+1032.535187854" watchObservedRunningTime="2026-11-01 20:15:44.818590956 +0000 UTC m=+1032.539669634"
Nov 01 20:15:45.809645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:45.809608    5041 generic.go:356] "Generic (PLEG): container finished" podID="1cf721fd-7f50-4aab-97a3-c9f756c9728d" containerID="91a84058d339bdb746042a8199d4ac67983530a99b5a92764311b401b42a9ede" exitCode=0
Nov 01 20:15:45.810067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:45.809672    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz" event={"ID":"1cf721fd-7f50-4aab-97a3-c9f756c9728d","Type":"ContainerDied","Data":"91a84058d339bdb746042a8199d4ac67983530a99b5a92764311b401b42a9ede"}
Nov 01 20:15:47.205545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.205506    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:47.291622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.291587    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\") pod \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") "
Nov 01 20:15:47.291822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.291754    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\") pod \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") "
Nov 01 20:15:47.291925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.291894    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\" (UniqueName: \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\") pod \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\" (UID: \"1cf721fd-7f50-4aab-97a3-c9f756c9728d\") "
Nov 01 20:15:47.292127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.292098    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca" pod "1cf721fd-7f50-4aab-97a3-c9f756c9728d" (UID: "1cf721fd-7f50-4aab-97a3-c9f756c9728d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:47.292510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.292480    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:47.293943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.293910    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca" pod "1cf721fd-7f50-4aab-97a3-c9f756c9728d" (UID: "1cf721fd-7f50-4aab-97a3-c9f756c9728d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:47.294067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.294044    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8" pod "1cf721fd-7f50-4aab-97a3-c9f756c9728d" (UID: "1cf721fd-7f50-4aab-97a3-c9f756c9728d"). InnerVolumeSpecName "kube-api-access-s7pn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:47.395079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.395036    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cf721fd-7f50-4aab-97a3-c9f756c9728d-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:47.395288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.395083    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7pn8\" (UniqueName: \"kubernetes.io/projected/1cf721fd-7f50-4aab-97a3-c9f756c9728d-kube-api-access-s7pn8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:47.841157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.841117    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz" event={"ID":"1cf721fd-7f50-4aab-97a3-c9f756c9728d","Type":"ContainerDied","Data":"919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660"}
Nov 01 20:15:47.841157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.841141    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"
Nov 01 20:15:47.841386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:47.841166    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919f093286883407ef42d1ce419bca3492ee17b81cfa245e9078a045686fb660"
Nov 01 20:15:51.831398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:51.831350    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"]
Nov 01 20:15:51.839038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:51.839005    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b54c8e3-n65nj"]
Nov 01 20:15:52.526625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:52.526580    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f916d1d5-a0dd-469f-aa88-a30f609ccec9" path="/var/lib/kubelet/pods/f916d1d5-a0dd-469f-aa88-a30f609ccec9/volumes"
Nov 01 20:15:53.025101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:53.025067    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"]
Nov 01 20:15:53.029518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:53.029492    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae47fd2-grd9b"]
Nov 01 20:15:54.022831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:54.022795    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"]
Nov 01 20:15:54.029193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:54.029156    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b915099b-f567n"]
Nov 01 20:15:54.429335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:54.429301    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77494a1d-0d8d-4c26-a375-962e712d5717" path="/var/lib/kubelet/pods/77494a1d-0d8d-4c26-a375-962e712d5717/volumes"
Nov 01 20:15:54.429921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:54.429901    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36e8776-861e-45d7-9f88-0a61c3a4eadb" path="/var/lib/kubelet/pods/d36e8776-861e-45d7-9f88-0a61c3a4eadb/volumes"
Nov 01 20:15:55.057551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.057517    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"]
Nov 01 20:15:55.058373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.058352    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1cf721fd-7f50-4aab-97a3-c9f756c9728d" containerName="revoke-keys"
Nov 01 20:15:55.058373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.058373    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf721fd-7f50-4aab-97a3-c9f756c9728d" containerName="revoke-keys"
Nov 01 20:15:55.058514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.058488    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf721fd-7f50-4aab-97a3-c9f756c9728d" containerName="revoke-keys"
Nov 01 20:15:55.063452 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.063429    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.068339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.068315    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:15:55.068522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.068339    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:15:55.079976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.079949    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:15:55.080159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.080047    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:15:55.084063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.083865    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"]
Nov 01 20:15:55.182872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.182835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.183032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.182984    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.183032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.183007    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7w7\" (UniqueName: \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.299799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.284677    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.299799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.284896    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.299799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.284935    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7w7\" (UniqueName: \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.299799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.287681    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.301557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.301522    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7w7\" (UniqueName: \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.303565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.303541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e4d566c-866ss\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.386911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.386835    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:55.703534 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:15:55.703502    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6903c179_0d1b_46fd_ab90_898741d8cd3a.slice/crio-6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283 WatchSource:0}: Error finding container 6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283: Status 404 returned error can't find the container with id 6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283
Nov 01 20:15:55.704472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.704443    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"]
Nov 01 20:15:55.960426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:55.960343    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss" event={"ID":"6903c179-0d1b-46fd-ab90-898741d8cd3a","Type":"ContainerStarted","Data":"6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283"}
Nov 01 20:15:56.970155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:56.970124    5041 generic.go:356] "Generic (PLEG): container finished" podID="6903c179-0d1b-46fd-ab90-898741d8cd3a" containerID="5fc904696ddab961fdb5c6bc0a4c6e3e972e842f8679eabf25a73531c4c89495" exitCode=0
Nov 01 20:15:56.970572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:56.970212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss" event={"ID":"6903c179-0d1b-46fd-ab90-898741d8cd3a","Type":"ContainerDied","Data":"5fc904696ddab961fdb5c6bc0a4c6e3e972e842f8679eabf25a73531c4c89495"}
Nov 01 20:15:58.241828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.241720    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:58.364280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.364237    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\" (UniqueName: \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\") pod \"6903c179-0d1b-46fd-ab90-898741d8cd3a\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") "
Nov 01 20:15:58.364486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.364465    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\") pod \"6903c179-0d1b-46fd-ab90-898741d8cd3a\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") "
Nov 01 20:15:58.364560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.364531    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\") pod \"6903c179-0d1b-46fd-ab90-898741d8cd3a\" (UID: \"6903c179-0d1b-46fd-ab90-898741d8cd3a\") "
Nov 01 20:15:58.364901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.364877    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca" pod "6903c179-0d1b-46fd-ab90-898741d8cd3a" (UID: "6903c179-0d1b-46fd-ab90-898741d8cd3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:15:58.365055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.365036    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.366464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.366445    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7" pod "6903c179-0d1b-46fd-ab90-898741d8cd3a" (UID: "6903c179-0d1b-46fd-ab90-898741d8cd3a"). InnerVolumeSpecName "kube-api-access-zf7w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:15:58.366636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.366617    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca" pod "6903c179-0d1b-46fd-ab90-898741d8cd3a" (UID: "6903c179-0d1b-46fd-ab90-898741d8cd3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:15:58.473027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.472990    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7w7\" (UniqueName: \"kubernetes.io/projected/6903c179-0d1b-46fd-ab90-898741d8cd3a-kube-api-access-zf7w7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.473271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:58.473253    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6903c179-0d1b-46fd-ab90-898741d8cd3a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:15:58.587391 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:15:58.587353    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:15:59.012927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.012903    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"
Nov 01 20:15:59.013315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.013025    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss" event={"ID":"6903c179-0d1b-46fd-ab90-898741d8cd3a","Type":"ContainerDied","Data":"6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283"}
Nov 01 20:15:59.013426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.013346    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6246286c472a0469f5c45893fc09d0e0e8713fd3311c2ac48ae7a5deac6a8283"
Nov 01 20:15:59.615801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.611100    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:15:59.615801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.612743    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6903c179-0d1b-46fd-ab90-898741d8cd3a" containerName="revoke-keys"
Nov 01 20:15:59.615801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.612769    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6903c179-0d1b-46fd-ab90-898741d8cd3a" containerName="revoke-keys"
Nov 01 20:15:59.654322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.623076    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6903c179-0d1b-46fd-ab90-898741d8cd3a" containerName="revoke-keys"
Nov 01 20:15:59.707868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.705382    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.718801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.711050    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9de2ac-openshift-default-dockercfg-4kqbj\""
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935712    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935767    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935832    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935894    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935924    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.935964    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.936057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.936086    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:15:59.972858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:15:59.936211    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.027223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.027193    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.101983    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126411    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126555    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126641    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126675    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126727    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126835    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126872    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.126918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.130803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.102490    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.183212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.139265    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.183212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.140295    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.183212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.142988    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.183212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.143323    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.228870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.228793    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.314802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.308382    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.314802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.309044    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.332803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.320344    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:00.339333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:00.338007    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:01.743998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:01.734636    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:01.743998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:01.734733    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:01.753629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:01.753574    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:01.892991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:01.892938    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:16:02.436213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:02.436120    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerStarted","Data":"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"}
Nov 01 20:16:02.436213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:02.436187    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerStarted","Data":"b77783bbd8ba3d2fe6d00e83aabaa0cccde27558abb27db507be7ea08a6a5085"}
Nov 01 20:16:02.455949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:02.455887    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podStartSLOduration=3.455873042 podStartE2EDuration="3.455873042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:02.454304855 +0000 UTC m=+1050.175383542" watchObservedRunningTime="2026-11-01 20:16:02.455873042 +0000 UTC m=+1050.176951717"
Nov 01 20:16:03.338471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:03.338434    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:03.345219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:03.345191    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:03.445324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:03.445290    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:03.446219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:03.446193    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:04.038051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:04.038016    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"]
Nov 01 20:16:04.056118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:04.056064    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c5b5b-lm5lj"]
Nov 01 20:16:04.429504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:04.429417    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4f1503-9f5b-44be-8ad3-325118ba45d3" path="/var/lib/kubelet/pods/ac4f1503-9f5b-44be-8ad3-325118ba45d3/volumes"
Nov 01 20:16:08.037065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.037031    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"]
Nov 01 20:16:08.042155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.042126    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"]
Nov 01 20:16:08.048986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.048958    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b07fe65-mzj5w"]
Nov 01 20:16:08.054853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.054820    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7dd1877-nw7lf"]
Nov 01 20:16:08.428928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.428849    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f548acd-ed33-4e54-a418-59f5bb610eaa" path="/var/lib/kubelet/pods/4f548acd-ed33-4e54-a418-59f5bb610eaa/volumes"
Nov 01 20:16:08.429396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:08.429375    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea89c80-951d-40f0-b059-7475a699d1ae" path="/var/lib/kubelet/pods/eea89c80-951d-40f0-b059-7475a699d1ae/volumes"
Nov 01 20:16:17.706159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.706125    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"]
Nov 01 20:16:17.712695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.712668    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.718496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.718472    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"]
Nov 01 20:16:17.719112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.719083    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:16:17.719458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.719430    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:16:17.719583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.719493    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:16:17.719583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.719431    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:16:17.870291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.870255    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.870536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.870510    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.870809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.870767    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k766m\" (UniqueName: \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.972937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.972829    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.972937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.972907    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.973258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.973090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k766m\" (UniqueName: \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.975127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.975083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.976959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.976930    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:17.987294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:17.987152    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k766m\" (UniqueName: \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\") pod \"maas-api-revoke-keys-e2e-derive-ea421713-bvz7g\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:18.044612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:18.044572    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:18.363875 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:16:18.363743    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b20ccdc_ff63_4477_b8c1_1e07efcb7d74.slice/crio-f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d WatchSource:0}: Error finding container f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d: Status 404 returned error can't find the container with id f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d
Nov 01 20:16:18.367044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:18.367020    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"]
Nov 01 20:16:18.666953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:18.666922    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g" event={"ID":"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74","Type":"ContainerStarted","Data":"f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d"}
Nov 01 20:16:19.026231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:19.026140    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"]
Nov 01 20:16:19.031304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:19.031274    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f9c938c-6qhq7"]
Nov 01 20:16:19.677745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:19.677715    5041 generic.go:356] "Generic (PLEG): container finished" podID="4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" containerID="237c8c0d55b87ed0f6fee808e2432ee211a47cfaa1a8a619a3d4cf1ecf25522b" exitCode=0
Nov 01 20:16:19.677961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:19.677811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g" event={"ID":"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74","Type":"ContainerDied","Data":"237c8c0d55b87ed0f6fee808e2432ee211a47cfaa1a8a619a3d4cf1ecf25522b"}
Nov 01 20:16:20.033074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.033042    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"]
Nov 01 20:16:20.037124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.037102    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9122cc42-259ll"]
Nov 01 20:16:20.446265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.446221    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a0aa61-11b3-405f-a19c-c51e9c616272" path="/var/lib/kubelet/pods/b5a0aa61-11b3-405f-a19c-c51e9c616272/volumes"
Nov 01 20:16:20.446849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.446829    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4793189-93dc-4630-a127-bd5d9ea5fd45" path="/var/lib/kubelet/pods/c4793189-93dc-4630-a127-bd5d9ea5fd45/volumes"
Nov 01 20:16:20.870063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.870035    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:20.943850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.934333    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\") pod \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") "
Nov 01 20:16:20.943850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.934435    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\") pod \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") "
Nov 01 20:16:20.943850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.934598    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\" (UniqueName: \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\") pod \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\" (UID: \"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74\") "
Nov 01 20:16:20.943850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.937975    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca" pod "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" (UID: "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:16:20.966799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.949405    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m" pod "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" (UID: "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74"). InnerVolumeSpecName "kube-api-access-k766m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:16:20.970271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:20.970227    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca" pod "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" (UID: "4b20ccdc-ff63-4477-b8c1-1e07efcb7d74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:16:21.037501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.037455    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:21.038064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.038040    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:21.038528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.038508    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k766m\" (UniqueName: \"kubernetes.io/projected/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74-kube-api-access-k766m\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:16:21.437656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.437577    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:16:21.438661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.438634    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" containerName="revoke-keys"
Nov 01 20:16:21.438661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.438663    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" containerName="revoke-keys"
Nov 01 20:16:21.438945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.438863    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" containerName="revoke-keys"
Nov 01 20:16:21.443307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.443286    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.459987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.459953    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:16:21.462122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.462093    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-9be196-openshift-default-dockercfg-wrldh\""
Nov 01 20:16:21.562861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.562830    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.562874    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.562966    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6p6\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563016    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563092    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563157    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563211    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563247    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.563293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.563265    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670473    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670549    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6p6\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670618    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670652    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670717    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.670803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670766    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.671296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670833    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.671296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.670860    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.671296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.671030    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.671529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.671495    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.681122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.671799    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.681122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.672981    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.681122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.673522    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.681122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.674443    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.705877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.705727    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.707822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.706686    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6p6\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.708177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.708150    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.708442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.708427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\") pod \"e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") " pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:21.733562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.733517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g" event={"ID":"4b20ccdc-ff63-4477-b8c1-1e07efcb7d74","Type":"ContainerDied","Data":"f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d"}
Nov 01 20:16:21.733842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.733824    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47ef6e6ddec15d3e7c4e025a1fcf7ca4ab1749878154ea5161efef3ab0bb71d"
Nov 01 20:16:21.734116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.734102    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"
Nov 01 20:16:21.774613 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:21.774325    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:22.024265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.024181    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"]
Nov 01 20:16:22.028680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.028654    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481c79e3-bmtlw"]
Nov 01 20:16:22.069493 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:16:22.069462    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1a442e_4b5e_43af_be3f_629d9b16aa70.slice/crio-217412f2d92a5de9c5bd0de92c38ce1f3ca8f87b44b2be2da2a2ce82d1733003 WatchSource:0}: Error finding container 217412f2d92a5de9c5bd0de92c38ce1f3ca8f87b44b2be2da2a2ce82d1733003: Status 404 returned error can't find the container with id 217412f2d92a5de9c5bd0de92c38ce1f3ca8f87b44b2be2da2a2ce82d1733003
Nov 01 20:16:22.070367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.070343    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:16:22.072161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.072127    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:22.072267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.072204    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:22.072267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.072252    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:22.433108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.433072    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3c51f4-5256-48d6-999d-c3aef4a2e710" path="/var/lib/kubelet/pods/1a3c51f4-5256-48d6-999d-c3aef4a2e710/volumes"
Nov 01 20:16:22.743989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.743899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerStarted","Data":"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"}
Nov 01 20:16:22.743989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.743962    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerStarted","Data":"217412f2d92a5de9c5bd0de92c38ce1f3ca8f87b44b2be2da2a2ce82d1733003"}
Nov 01 20:16:22.767168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.767116    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podStartSLOduration=1.767103655 podStartE2EDuration="1.767103655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:22.763709037 +0000 UTC m=+1070.484787713" watchObservedRunningTime="2026-11-01 20:16:22.767103655 +0000 UTC m=+1070.488182330"
Nov 01 20:16:22.776163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:22.776134    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:23.782119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:23.782090    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:24.760941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:24.760907    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:24.761935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:24.761910    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:27.317431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.317390    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:16:27.326118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.326089    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.328762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.328739    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:16:27.334591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.334566    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"default-dockercfg-8w4sw\""
Nov 01 20:16:27.334591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.334583    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"simulator-w3-9de2ac-kserve-self-signed-certs\""
Nov 01 20:16:27.334837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.334630    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"kube-root-ca.crt\""
Nov 01 20:16:27.336531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.336303    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9de2ac\"/\"openshift-service-ca.crt\""
Nov 01 20:16:27.388355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388325    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.388560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388385    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.388560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388492    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8w8z\" (UniqueName: \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.388692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388626    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.388692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388664    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.388839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.388819    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.495757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.495714    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.495979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.495837    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.495979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.495876    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8w8z\" (UniqueName: \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.495979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.495901    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.495979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.495926    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.496162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.496077    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.496276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.496252    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.496564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.496541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.497130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.497108    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.505376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.505350    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.509633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.509559    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8w8z\" (UniqueName: \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.512045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.512018    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\") pod \"simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") " pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.643417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.643336    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:27.941038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:27.941009    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:16:27.941260 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:16:27.941237    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcded9ee_c2f8_4780_9234_705481fb1333.slice/crio-c4e3a58e6f3f5320fe2c9d40b7e763b35abc24be6c6471cbeb5ac68b299fbb06 WatchSource:0}: Error finding container c4e3a58e6f3f5320fe2c9d40b7e763b35abc24be6c6471cbeb5ac68b299fbb06: Status 404 returned error can't find the container with id c4e3a58e6f3f5320fe2c9d40b7e763b35abc24be6c6471cbeb5ac68b299fbb06
Nov 01 20:16:28.543620 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:16:28.543574    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:16:28.831285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.831188    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" event={"ID":"bcded9ee-c2f8-4780-9234-705481fb1333","Type":"ContainerStarted","Data":"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"}
Nov 01 20:16:28.832436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.832402    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" event={"ID":"bcded9ee-c2f8-4780-9234-705481fb1333","Type":"ContainerStarted","Data":"c4e3a58e6f3f5320fe2c9d40b7e763b35abc24be6c6471cbeb5ac68b299fbb06"}
Nov 01 20:16:28.854102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.854049    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podStartSLOduration=1.8540342380000001 podStartE2EDuration="1.854034238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:28.847646292 +0000 UTC m=+1076.568725349" watchObservedRunningTime="2026-11-01 20:16:28.854034238 +0000 UTC m=+1076.575112914"
Nov 01 20:16:28.955129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.955093    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:16:28.959904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.959880    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:28.972846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:28.972819    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064320    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064472    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064504    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064531    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064670    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj79\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064705    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064744    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.065800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.064804    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj79\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169101    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169147    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169283    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169513    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169551    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.169578    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.170874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.170169    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.175298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.174523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.175298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.174846    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.175298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.175109    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.175824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.175761    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.176432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.176409    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.187503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.185347    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj79\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.187503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.187295    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.198685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.198635    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.278715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.278679    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:29.580742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.580702    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:16:29.581231 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:16:29.580896    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddffed5_308a_461f_bd60_1ce63aa6b4ba.slice/crio-5aa858ba6e752df70c2b59bf237a45bd9ae8993ff7047d6813a25bf0fdce392b WatchSource:0}: Error finding container 5aa858ba6e752df70c2b59bf237a45bd9ae8993ff7047d6813a25bf0fdce392b: Status 404 returned error can't find the container with id 5aa858ba6e752df70c2b59bf237a45bd9ae8993ff7047d6813a25bf0fdce392b
Nov 01 20:16:29.583572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.583531    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:29.583689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.583601    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:29.583689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.583631    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:29.644047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.644023    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:29.895856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.895517    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" event={"ID":"bddffed5-308a-461f-bd60-1ce63aa6b4ba","Type":"ContainerStarted","Data":"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"}
Nov 01 20:16:29.895856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.895576    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" event={"ID":"bddffed5-308a-461f-bd60-1ce63aa6b4ba","Type":"ContainerStarted","Data":"5aa858ba6e752df70c2b59bf237a45bd9ae8993ff7047d6813a25bf0fdce392b"}
Nov 01 20:16:29.946862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:29.946796    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podStartSLOduration=1.946737922 podStartE2EDuration="1.946737922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:29.940525256 +0000 UTC m=+1077.661603937" watchObservedRunningTime="2026-11-01 20:16:29.946737922 +0000 UTC m=+1077.667816598"
Nov 01 20:16:30.284876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:30.282848    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:30.284876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:30.282982    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:30.284876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:30.283028    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:31.286802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:31.286510    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:31.286802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:31.286568    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:31.426410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:31.426378    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"]
Nov 01 20:16:31.430897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:31.430870    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ea421713-bvz7g"]
Nov 01 20:16:32.290804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:32.286220    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:32.290804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:32.286281    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:32.430876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:32.430766    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b20ccdc-ff63-4477-b8c1-1e07efcb7d74" path="/var/lib/kubelet/pods/4b20ccdc-ff63-4477-b8c1-1e07efcb7d74/volumes"
Nov 01 20:16:33.293803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:33.286204    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:33.293803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:33.286265    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:33.909198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:33.909170    5041 scope.go:122] "RemoveContainer" containerID="d077fb66afd4dc06a66cdc528c656437600f5064be624274bb06e6ad3fc247ae"
Nov 01 20:16:34.120006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.107946    5041 scope.go:122] "RemoveContainer" containerID="d28531e26af7bb1b3cc09472a678ad70f27c9e340f34662c8065219c58379e32"
Nov 01 20:16:34.286808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.281885    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:34.286808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.281948    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:34.319206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.319124    5041 scope.go:122] "RemoveContainer" containerID="684b4c106abda7bb0a36c12c1f98a18fce554de842e0c24558889526f94da909"
Nov 01 20:16:34.337881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.337859    5041 scope.go:122] "RemoveContainer" containerID="68ec18dddf24ecce39afc5cc0738953c9254e0b6911988754a2232ab56204e0a"
Nov 01 20:16:34.407110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.407082    5041 scope.go:122] "RemoveContainer" containerID="ca9cee8ef80f92087d867a39d2678b0e37032d68860280521cd3e6bc87cc5346"
Nov 01 20:16:34.418819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.418764    5041 scope.go:122] "RemoveContainer" containerID="10dac9d7d9cf6d5dd4104d7d840a78f15222e0b3f109f7f90be89a9513e182fa"
Nov 01 20:16:34.432538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.432511    5041 scope.go:122] "RemoveContainer" containerID="b427b06385e2ccd416cb103a05d5a236b0af4fd5c4aeeb6b56918b4338e40e35"
Nov 01 20:16:34.474806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:34.473002    5041 scope.go:122] "RemoveContainer" containerID="1a31533bf27773582aab6aaddd882c9e08961bea81ef52d986bd7fbebeb0f37e"
Nov 01 20:16:35.025498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:35.025464    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"]
Nov 01 20:16:35.030382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:35.030354    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2728798a-fk7vw"]
Nov 01 20:16:35.296442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:35.296403    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:35.296611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:35.296502    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:36.279287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:36.279254    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 01 20:16:36.279702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:36.279308    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 01 20:16:36.430523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:36.430443    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0affbdbf-051b-410f-aee1-1e2fc41643b2" path="/var/lib/kubelet/pods/0affbdbf-051b-410f-aee1-1e2fc41643b2/volumes"
Nov 01 20:16:37.023216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.023181    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"]
Nov 01 20:16:37.027685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.027656    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2a433-b7rvp"]
Nov 01 20:16:37.285536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.285508    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:37.286051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.286025    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:37.286897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.286872    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:16:37.644000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.643912    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:37.661227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:37.661201    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:38.300638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:38.300604    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:16:38.447077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:38.447026    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4822bd-1d0c-43d8-9ec0-7d133866815b" path="/var/lib/kubelet/pods/fe4822bd-1d0c-43d8-9ec0-7d133866815b/volumes"
Nov 01 20:16:39.027487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:39.027455    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"]
Nov 01 20:16:39.032740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:39.032706    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7674458c-s69z9"]
Nov 01 20:16:40.430444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:40.430413    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9439cd2-366b-42c4-a049-e17117e6ea74" path="/var/lib/kubelet/pods/a9439cd2-366b-42c4-a049-e17117e6ea74/volumes"
Nov 01 20:16:41.312151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.312115    5041 generic.go:356] "Generic (PLEG): container finished" podID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerID="172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39" exitCode=0
Nov 01 20:16:41.312407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.312191    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerDied","Data":"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"}
Nov 01 20:16:41.312664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.312646    5041 scope.go:122] "RemoveContainer" containerID="172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"
Nov 01 20:16:41.314327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.314289    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:41.314421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.314360    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:41.314421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:41.314391    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:42.328864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:42.328823    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerStarted","Data":"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"}
Nov 01 20:16:42.338838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:42.338815    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:42.339016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:42.338997    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:42.339065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:42.339038    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:43.348799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:43.340168    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:43.348799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:43.340267    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:44.339113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:44.338949    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:44.339113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:44.339023    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:45.339352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.339313    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:45.339862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.339374    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:45.806736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.799584    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:16:45.810803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.808739    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:45.815879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.814988    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:16:45.815879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:45.815562    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e83b4b-openshift-default-dockercfg-xsl96\""
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045207    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045276    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045330    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045367    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045487    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045581    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045654    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.047805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.045723    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwqv\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.073853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.072000    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.177975    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178054    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178094    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178339    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178426    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178507    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwqv\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178548    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.178804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.178573    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.179470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.179081    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.179470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.179458    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.182147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.182115    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.182441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.182416    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.182536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.182419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.185032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.185009    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.185160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.185016    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.191400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.191378    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.191494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.191471    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwqv\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\") pod \"e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") " pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.229209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.229184    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:46.339137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.339058    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:46.339286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.339133    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:46.632181 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:16:46.632149    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40b3d08_2017_42dc_bfca_c5f7a9b04949.slice/crio-dc2e26817ddd0d9fc6971993bbf6604be5be0d32a9e4e26a926b7881c438abd5 WatchSource:0}: Error finding container dc2e26817ddd0d9fc6971993bbf6604be5be0d32a9e4e26a926b7881c438abd5: Status 404 returned error can't find the container with id dc2e26817ddd0d9fc6971993bbf6604be5be0d32a9e4e26a926b7881c438abd5
Nov 01 20:16:46.632596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.632323    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:16:46.634736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.634701    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:46.634882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.634803    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:46.634882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:46.634848    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:47.339067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:47.339028    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 01 20:16:47.339270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:47.339087    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 01 20:16:47.406647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:47.406608    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerStarted","Data":"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"}
Nov 01 20:16:47.407154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:47.407118    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerStarted","Data":"dc2e26817ddd0d9fc6971993bbf6604be5be0d32a9e4e26a926b7881c438abd5"}
Nov 01 20:16:47.434297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:47.434240    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podStartSLOduration=2.434224784 podStartE2EDuration="2.434224784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:47.429997865 +0000 UTC m=+1095.151076542" watchObservedRunningTime="2026-11-01 20:16:47.434224784 +0000 UTC m=+1095.155303460"
Nov 01 20:16:48.231860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.231828    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:48.265799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.265139    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:48.434801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.434153    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:48.436347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.436086    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:16:48.445160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.445131    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:48.931624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:48.931592    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:49.468709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:49.468023    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:16:50.091210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:50.091172    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"]
Nov 01 20:16:50.094578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:50.094549    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45216106-whqm2"]
Nov 01 20:16:50.429823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:50.429703    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96290b8e-b541-473a-9870-2c7a29eb9881" path="/var/lib/kubelet/pods/96290b8e-b541-473a-9870-2c7a29eb9881/volumes"
Nov 01 20:16:55.802953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:55.802908    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:16:55.803451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:55.803042    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:16:58.549596 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:16:58.549560    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:16:58.671118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.671087    5041 generic.go:356] "Generic (PLEG): container finished" podID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerID="7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3" exitCode=0
Nov 01 20:16:58.671290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.671168    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerDied","Data":"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"}
Nov 01 20:16:58.671615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.671602    5041 scope.go:122] "RemoveContainer" containerID="7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"
Nov 01 20:16:58.673252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.673223    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:58.673336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.673291    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:58.673336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.673324    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:58.874374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.874278    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:16:58.894845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.888528    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.894845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.894689    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:16:58.898387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.898351    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6f810f-openshift-default-dockercfg-7qw2j\""
Nov 01 20:16:58.966431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.966390    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.991283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.991235    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.994945    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995036    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995088    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995113    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995177    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2v4h\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995208    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:58.998904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:58.995262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.107745    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2v4h\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.107817    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.107870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108276    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108322    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.108346    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.109233    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.111801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.109870    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.206851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.181437    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.206851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.181699    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.206851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.182245    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.207730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.207663    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.231809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.210722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2v4h\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.231809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.224479    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.260150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.260113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\") pod \"e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") " pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.524733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.524652    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:16:59.693724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.692506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerStarted","Data":"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"}
Nov 01 20:16:59.776276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.776192    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:16:59.776438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.776393    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:16:59.776480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.776460    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:16:59.905817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.894068    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:16:59.928067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.926675    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:59.928067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.926825    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:16:59.928067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:16:59.926885    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:00.729063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:00.728877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerStarted","Data":"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"}
Nov 01 20:17:00.730012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:00.729119    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerStarted","Data":"6cd8f9739a7ca4a5321c4cb5bd76fea3c5db4c413e3a4d2c20ed9e29d4d122c1"}
Nov 01 20:17:00.760815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:00.760216    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podStartSLOduration=2.760195191 podStartE2EDuration="2.760195191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:00.758355742 +0000 UTC m=+1108.479434420" watchObservedRunningTime="2026-11-01 20:17:00.760195191 +0000 UTC m=+1108.481273870"
Nov 01 20:17:00.783956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:00.783919    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:17:00.785323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:00.785283    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:17:01.525825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.525767    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:01.531363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.531337    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:01.737338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.737304    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:01.738328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.738300    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:01.779403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.779309    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:17:01.779707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:01.779668    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:17:02.789800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:02.777286    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:17:02.789800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:02.777348    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:17:03.076855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:03.075849    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"]
Nov 01 20:17:03.090209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:03.090165    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5625be3c-dqnh5"]
Nov 01 20:17:03.780531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:03.779246    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:17:03.780531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:03.779314    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:17:04.435158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:04.435122    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948b9504-de7d-46c9-a928-ad3e9680a183" path="/var/lib/kubelet/pods/948b9504-de7d-46c9-a928-ad3e9680a183/volumes"
Nov 01 20:17:04.776590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:04.776497    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 01 20:17:04.776590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:04.776566    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 01 20:17:05.911481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:05.911449    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:17:05.912109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:05.912033    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:17:05.913153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:05.913132    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:17:12.023007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:12.022977    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"]
Nov 01 20:17:12.026993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:12.026948    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-335aa786-vt495"]
Nov 01 20:17:12.433309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:12.433274    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7de724-87ff-4748-b6e7-4cde95898864" path="/var/lib/kubelet/pods/cd7de724-87ff-4748-b6e7-4cde95898864/volumes"
Nov 01 20:17:14.022317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:14.022229    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"]
Nov 01 20:17:14.026617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:14.026587    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f431ba-swdrt"]
Nov 01 20:17:14.430077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:14.429927    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9652689f-1204-450e-a9cb-a5901386796f" path="/var/lib/kubelet/pods/9652689f-1204-450e-a9cb-a5901386796f/volumes"
Nov 01 20:17:25.024369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:25.024320    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"]
Nov 01 20:17:25.028691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:25.028660    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48743828-hlk7x"]
Nov 01 20:17:26.492805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.491730    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ffe76f-fc73-4874-9f90-51c0eb0364f3" path="/var/lib/kubelet/pods/b3ffe76f-fc73-4874-9f90-51c0eb0364f3/volumes"
Nov 01 20:17:26.891545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.891507    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"]
Nov 01 20:17:26.909424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.909388    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:26.927807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.917052    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:17:26.927807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.920020    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:17:26.927807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.920103    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:17:26.927807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.920168    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:17:26.932836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.931360    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"]
Nov 01 20:17:26.986867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.986837    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:17:26.991139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.991110    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:17:26.991332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:26.991310    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy" containerID="cri-o://09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74" gracePeriod=30
Nov 01 20:17:27.087386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.087342    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrtn\" (UniqueName: \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.087579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.087492    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.087579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.087541    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.178644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.178545    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy" containerID="cri-o://c11b0459b751d25a85e7f37db496938ac8066fad37fe527384ddbddc78a2d364" gracePeriod=30
Nov 01 20:17:27.188601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.188570    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrtn\" (UniqueName: \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.188898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.188884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.189021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.189011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.189668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.189646    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.191159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.191137    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.195802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.195757    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrtn\" (UniqueName: \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\") pod \"maas-api-revoke-sub-9e97b7dc-bs4t7\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.281020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.280986    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:27.592835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:27.592661    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"]
Nov 01 20:17:28.039593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:28.035827    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"]
Nov 01 20:17:28.041671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:28.041609    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f811fc1d-8qjbz"]
Nov 01 20:17:28.365864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:28.363357    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7" event={"ID":"9fb687a5-8c6d-45ec-9840-15c582e4c58a","Type":"ContainerStarted","Data":"fb5e6493a00b315a8659b9e5dd5b51cfaa9c0ff9d0c4537dd15b3c87bde9523f"}
Nov 01 20:17:28.493805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:28.490434    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a25c60-f530-4989-8274-1ec0bb9d6ea5" path="/var/lib/kubelet/pods/78a25c60-f530-4989-8274-1ec0bb9d6ea5/volumes"
Nov 01 20:17:28.580116 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:17:28.579963    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:17:29.418175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:29.418087    5041 generic.go:356] "Generic (PLEG): container finished" podID="9fb687a5-8c6d-45ec-9840-15c582e4c58a" containerID="3689da22b997de044ba6f75ed65c517f0b130665f636e9f215c8b00ce13890bd" exitCode=0
Nov 01 20:17:29.418618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:29.418265    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7" event={"ID":"9fb687a5-8c6d-45ec-9840-15c582e4c58a","Type":"ContainerDied","Data":"3689da22b997de044ba6f75ed65c517f0b130665f636e9f215c8b00ce13890bd"}
Nov 01 20:17:30.618648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.618624    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:30.644386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.644348    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\") pod \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") "
Nov 01 20:17:30.644751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.644636    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\" (UniqueName: \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\") pod \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") "
Nov 01 20:17:30.644751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.644719    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\") pod \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\" (UID: \"9fb687a5-8c6d-45ec-9840-15c582e4c58a\") "
Nov 01 20:17:30.645822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.645794    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca" pod "9fb687a5-8c6d-45ec-9840-15c582e4c58a" (UID: "9fb687a5-8c6d-45ec-9840-15c582e4c58a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:30.654297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.654252    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca" pod "9fb687a5-8c6d-45ec-9840-15c582e4c58a" (UID: "9fb687a5-8c6d-45ec-9840-15c582e4c58a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:17:30.654467 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.654393    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn" pod "9fb687a5-8c6d-45ec-9840-15c582e4c58a" (UID: "9fb687a5-8c6d-45ec-9840-15c582e4c58a"). InnerVolumeSpecName "kube-api-access-zbrtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:30.747217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.747091    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:30.747217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.747133    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb687a5-8c6d-45ec-9840-15c582e4c58a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:30.747217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:30.747193    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbrtn\" (UniqueName: \"kubernetes.io/projected/9fb687a5-8c6d-45ec-9840-15c582e4c58a-kube-api-access-zbrtn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:31.446835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:31.446796    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7" event={"ID":"9fb687a5-8c6d-45ec-9840-15c582e4c58a","Type":"ContainerDied","Data":"fb5e6493a00b315a8659b9e5dd5b51cfaa9c0ff9d0c4537dd15b3c87bde9523f"}
Nov 01 20:17:31.447052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:31.446860    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5e6493a00b315a8659b9e5dd5b51cfaa9c0ff9d0c4537dd15b3c87bde9523f"
Nov 01 20:17:31.447052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:31.446889    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"
Nov 01 20:17:32.360732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.360696    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:17:32.460595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.460561    5041 generic.go:356] "Generic (PLEG): container finished" podID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerID="09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74" exitCode=0
Nov 01 20:17:32.460831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.460637    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"
Nov 01 20:17:32.460831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.460643    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" event={"ID":"1940b5aa-9298-4837-bff7-32abb2bda4b8","Type":"ContainerDied","Data":"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"}
Nov 01 20:17:32.460831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.460710    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl" event={"ID":"1940b5aa-9298-4837-bff7-32abb2bda4b8","Type":"ContainerDied","Data":"8674ea658c6b8591370a71177c473242fa70a613fe4815272c5eaaa9b550d88e"}
Nov 01 20:17:32.460831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.460733    5041 scope.go:122] "RemoveContainer" containerID="09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"
Nov 01 20:17:32.463212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.463191    5041 generic.go:356] "Generic (PLEG): container finished" podID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerID="c11b0459b751d25a85e7f37db496938ac8066fad37fe527384ddbddc78a2d364" exitCode=0
Nov 01 20:17:32.464428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.463406    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" event={"ID":"0c0fa21f-1a0c-4490-809d-2640b709504d","Type":"ContainerDied","Data":"c11b0459b751d25a85e7f37db496938ac8066fad37fe527384ddbddc78a2d364"}
Nov 01 20:17:32.480027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.479914    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.480027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.479989    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.480633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.480577    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481245    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481346    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481388    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481431    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481458    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481504    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.481965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.481622    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\") pod \"1940b5aa-9298-4837-bff7-32abb2bda4b8\" (UID: \"1940b5aa-9298-4837-bff7-32abb2bda4b8\") "
Nov 01 20:17:32.483300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.483280    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.483774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.483699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.483902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.483865    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.484743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.484109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.484743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.484261    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:32.484743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.484410    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.485881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.485854    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:32.487424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.487393    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "kube-api-access-n2wmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:32.488685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.488656    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo" pod "1940b5aa-9298-4837-bff7-32abb2bda4b8" (UID: "1940b5aa-9298-4837-bff7-32abb2bda4b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:17:32.504332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.504267    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:17:32.517729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.517192    5041 scope.go:122] "RemoveContainer" containerID="09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"
Nov 01 20:17:32.517942 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:17:32.517861    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74\": container with ID starting with 09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74 not found: ID does not exist" containerID="09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"
Nov 01 20:17:32.517942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.517902    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74"} err="failed to get container status \"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74\": rpc error: code = NotFound desc = could not find container \"09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74\": container with ID starting with 09b4f2e13d16be93b5ae2a3634fec13063104f1acc16ad8b29bebfcef9a1ef74 not found: ID does not exist"
Nov 01 20:17:32.584106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.583958    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584189    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584229    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584333    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584400    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584432    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584506    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584565    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.584607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.584591    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\") pod \"0c0fa21f-1a0c-4490-809d-2640b709504d\" (UID: \"0c0fa21f-1a0c-4490-809d-2640b709504d\") "
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585145    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585268    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585352    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585299    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1940b5aa-9298-4837-bff7-32abb2bda4b8-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585476    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wmd\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-kube-api-access-n2wmd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585496    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585513    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585528    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.585532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585543    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.586104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585557    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.586104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585571    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.586104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585585    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1940b5aa-9298-4837-bff7-32abb2bda4b8-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.586104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585876    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.586104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.585937    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.588169 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.588138    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "kube-api-access-9kcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:32.588289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.588247    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:17:32.588469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.588441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:32.589465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.589435    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo" pod "0c0fa21f-1a0c-4490-809d-2640b709504d" (UID: "0c0fa21f-1a0c-4490-809d-2640b709504d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:17:32.687079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687041    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687074    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcwd\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-kube-api-access-9kcwd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687085    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687094    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687103    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687113    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0fa21f-1a0c-4490-809d-2640b709504d-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687121    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0fa21f-1a0c-4490-809d-2640b709504d-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.687399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.687129    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0fa21f-1a0c-4490-809d-2640b709504d-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:32.778965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.778928    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:17:32.779145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.778983    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:17:32.789627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.789582    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:17:32.797116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:32.797079    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-cvstl"]
Nov 01 20:17:33.475480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.475450    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"
Nov 01 20:17:33.475925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.475491    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k" event={"ID":"0c0fa21f-1a0c-4490-809d-2640b709504d","Type":"ContainerDied","Data":"da032f514ab4ab383f4b269f93486ea4de933884a8148e125d8f123649be0867"}
Nov 01 20:17:33.475925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.475567    5041 scope.go:122] "RemoveContainer" containerID="c11b0459b751d25a85e7f37db496938ac8066fad37fe527384ddbddc78a2d364"
Nov 01 20:17:33.501112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.501087    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:17:33.511506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.511475    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-fpw8k"]
Nov 01 20:17:33.566907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.566866    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:17:33.567953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.567924    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy"
Nov 01 20:17:33.567953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.567951    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy"
Nov 01 20:17:33.568159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.567986    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9fb687a5-8c6d-45ec-9840-15c582e4c58a" containerName="revoke-keys"
Nov 01 20:17:33.568159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.567994    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb687a5-8c6d-45ec-9840-15c582e4c58a" containerName="revoke-keys"
Nov 01 20:17:33.568159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.568035    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy"
Nov 01 20:17:33.568159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.568042    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy"
Nov 01 20:17:33.568417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.568210    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb687a5-8c6d-45ec-9840-15c582e4c58a" containerName="revoke-keys"
Nov 01 20:17:33.568417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.568230    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" containerName="istio-proxy"
Nov 01 20:17:33.568417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.568245    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" containerName="istio-proxy"
Nov 01 20:17:33.573951 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.573922    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.575733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.575701    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:17:33.579552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.579521    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e83b4b\"/\"kube-root-ca.crt\""
Nov 01 20:17:33.579656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.579628    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e83b4b\"/\"auth-test-model-e83b4b-kserve-self-signed-certs\""
Nov 01 20:17:33.579713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.579701    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-e83b4b\"/\"default-dockercfg-k8l84\""
Nov 01 20:17:33.579768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.579757    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-e83b4b\"/\"openshift-service-ca.crt\""
Nov 01 20:17:33.701514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701483    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.709968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701536    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzztl\" (UniqueName: \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.709968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701682    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.709968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701810    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.709968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.709968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.701946    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.803764    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.805892    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzztl\" (UniqueName: \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.806031    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.806146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.806238    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.807801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.806293    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.823280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.819251    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.823280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.822083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.823280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.822220    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.839802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.835064    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.854181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.853467    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.854181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.854123    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzztl\" (UniqueName: \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\") pod \"auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") " pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:33.896189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:33.895631    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:34.195028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.194998    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:17:34.195443 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:17:34.195421    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee8fd09e_a178_4f2c_abfb_cbe568d2e489.slice/crio-213c5f1b8ff93cfe95f7b9f1d6389d72e4ace6533f1f6f188215e6c9dc8b66e0 WatchSource:0}: Error finding container 213c5f1b8ff93cfe95f7b9f1d6389d72e4ace6533f1f6f188215e6c9dc8b66e0: Status 404 returned error can't find the container with id 213c5f1b8ff93cfe95f7b9f1d6389d72e4ace6533f1f6f188215e6c9dc8b66e0
Nov 01 20:17:34.430272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.430238    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0fa21f-1a0c-4490-809d-2640b709504d" path="/var/lib/kubelet/pods/0c0fa21f-1a0c-4490-809d-2640b709504d/volumes"
Nov 01 20:17:34.430877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.430851    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1940b5aa-9298-4837-bff7-32abb2bda4b8" path="/var/lib/kubelet/pods/1940b5aa-9298-4837-bff7-32abb2bda4b8/volumes"
Nov 01 20:17:34.484991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.484906    5041 generic.go:356] "Generic (PLEG): container finished" podID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerID="39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4" exitCode=0
Nov 01 20:17:34.484991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.484975    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerDied","Data":"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"}
Nov 01 20:17:34.485492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.485435    5041 scope.go:122] "RemoveContainer" containerID="39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"
Nov 01 20:17:34.486847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.486824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" event={"ID":"ee8fd09e-a178-4f2c-abfb-cbe568d2e489","Type":"ContainerStarted","Data":"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"}
Nov 01 20:17:34.486955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.486859    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" event={"ID":"ee8fd09e-a178-4f2c-abfb-cbe568d2e489","Type":"ContainerStarted","Data":"213c5f1b8ff93cfe95f7b9f1d6389d72e4ace6533f1f6f188215e6c9dc8b66e0"}
Nov 01 20:17:34.487201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.487167    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:34.487284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.487238    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:34.487284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.487278    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:34.533036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.531064    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podStartSLOduration=1.531046218 podStartE2EDuration="1.531046218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:34.524661253 +0000 UTC m=+1142.245739929" watchObservedRunningTime="2026-11-01 20:17:34.531046218 +0000 UTC m=+1142.252124895"
Nov 01 20:17:34.659696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.659661    5041 scope.go:122] "RemoveContainer" containerID="18f3ee07085ff0fad61792ab90bfb4bb93942b83f90a31741a6876fc6e9f949f"
Nov 01 20:17:34.689464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.687386    5041 scope.go:122] "RemoveContainer" containerID="aa87ca4c9ac46ae1da9384c4262ffabb47d4fac343edab5d9f498f29c3587b55"
Nov 01 20:17:34.712226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.712129    5041 scope.go:122] "RemoveContainer" containerID="6b3d745b09ee05316ff7def291f9f81a1932ceb99edd1751728df8c1ca427161"
Nov 01 20:17:34.726048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.723646    5041 scope.go:122] "RemoveContainer" containerID="b554b04925dfb12a38512cde9eaf2897e360f114d2bff114b784c034ef25ef6f"
Nov 01 20:17:34.734537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.734431    5041 scope.go:122] "RemoveContainer" containerID="fde913bfaa275b2062d72f7ab8833b82d2f1df9dca495dd767f12a4607878ddf"
Nov 01 20:17:34.746712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.746669    5041 scope.go:122] "RemoveContainer" containerID="417b9f560106ce31b6917aef42cbcfcd23dc23724b6c6044da9bc9e0f0896e47"
Nov 01 20:17:34.760225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.760202    5041 scope.go:122] "RemoveContainer" containerID="6c94cfd7a51f0ba58054c49318959e7b4379c99d8b701e43f0a4fddd1845a65d"
Nov 01 20:17:34.777105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.777079    5041 scope.go:122] "RemoveContainer" containerID="cb49e776215f8bcd740b043c0f5f8500a65d067f808ffff79534ebc846ca009a"
Nov 01 20:17:34.787935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.787910    5041 scope.go:122] "RemoveContainer" containerID="cda4f9604b80a5f5cb54d3a276602f785ec69cc497238d3fb3d6e5b531abf6cf"
Nov 01 20:17:34.800772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.800411    5041 scope.go:122] "RemoveContainer" containerID="0dc1c84b82a163a3291a2a317c1ca3b70a3bf44b739ba9758875f92e62c35344"
Nov 01 20:17:34.811954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.811929    5041 scope.go:122] "RemoveContainer" containerID="32adb2872416cf06c22ad5aaf6d4ce5a3c97e94afc051542b0af42bd19de5b84"
Nov 01 20:17:34.822679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.822655    5041 scope.go:122] "RemoveContainer" containerID="1a65f1032d6f79c48ed6cd1a76768da9e591f5172bed14e9d3f49805b6c35574"
Nov 01 20:17:34.897011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:34.896977    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:35.503176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:35.503145    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerStarted","Data":"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"}
Nov 01 20:17:35.529839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:35.524829    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:35.529839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:35.525005    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:35.529839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:35.525059    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:36.525836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:36.525803    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:36.526280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:36.525855    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:37.487848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.487807    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"]
Nov 01 20:17:37.496458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.496428    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.500257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.500229    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"]
Nov 01 20:17:37.500486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.500465    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:17:37.503289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.501600    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:17:37.503289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.501714    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:17:37.503289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.501805    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:17:37.504988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.504737    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq6x\" (UniqueName: \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.504988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.504821    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.504988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.504882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.525306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.525278    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:37.525436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.525320    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:37.605763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.605731    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq6x\" (UniqueName: \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.606250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.605808    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.606250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.605850    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.606538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.606517    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.608252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.608227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.614960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.614935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq6x\" (UniqueName: \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\") pod \"maas-api-revoke-sub-fc34296f-mclzl\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:37.823759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:37.823726    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:38.023094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.023032    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"]
Nov 01 20:17:38.027316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.027287    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9309b76-cbrtc"]
Nov 01 20:17:38.139102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.139075    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"]
Nov 01 20:17:38.139819 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:17:38.139788    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9336bd_7580_4b3e_9850_13ade5130ce1.slice/crio-339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5 WatchSource:0}: Error finding container 339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5: Status 404 returned error can't find the container with id 339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5
Nov 01 20:17:38.429310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.429232    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317db877-c521-49d5-bf59-c75e32634333" path="/var/lib/kubelet/pods/317db877-c521-49d5-bf59-c75e32634333/volumes"
Nov 01 20:17:38.525998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.525961    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:38.526151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.526022    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:38.535118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:38.535087    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl" event={"ID":"fd9336bd-7580-4b3e-9850-13ade5130ce1","Type":"ContainerStarted","Data":"339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5"}
Nov 01 20:17:39.036564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.036504    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"]
Nov 01 20:17:39.054742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.054692    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-540f116a-hzw6c"]
Nov 01 20:17:39.525477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.525397    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:39.525477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.525465    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:39.569271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.569232    5041 generic.go:356] "Generic (PLEG): container finished" podID="fd9336bd-7580-4b3e-9850-13ade5130ce1" containerID="fd09206139d2813e8a3c67a2a279fca993a86af78196aeb2c2b366b040578bc2" exitCode=0
Nov 01 20:17:39.569464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:39.569297    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl" event={"ID":"fd9336bd-7580-4b3e-9850-13ade5130ce1","Type":"ContainerDied","Data":"fd09206139d2813e8a3c67a2a279fca993a86af78196aeb2c2b366b040578bc2"}
Nov 01 20:17:40.440766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.440725    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04c4ff0-a311-4482-9cc2-332b0e8b409d" path="/var/lib/kubelet/pods/e04c4ff0-a311-4482-9cc2-332b0e8b409d/volumes"
Nov 01 20:17:40.525964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.525928    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Nov 01 20:17:40.526162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.526027    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Nov 01 20:17:40.761403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.761370    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:40.894455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.894426    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\") pod \"fd9336bd-7580-4b3e-9850-13ade5130ce1\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") "
Nov 01 20:17:40.894626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.894583    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\" (UniqueName: \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\") pod \"fd9336bd-7580-4b3e-9850-13ade5130ce1\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") "
Nov 01 20:17:40.894696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.894680    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\") pod \"fd9336bd-7580-4b3e-9850-13ade5130ce1\" (UID: \"fd9336bd-7580-4b3e-9850-13ade5130ce1\") "
Nov 01 20:17:40.894875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.894846    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca" pod "fd9336bd-7580-4b3e-9850-13ade5130ce1" (UID: "fd9336bd-7580-4b3e-9850-13ade5130ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:40.895289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.895269    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:40.896682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.896654    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x" pod "fd9336bd-7580-4b3e-9850-13ade5130ce1" (UID: "fd9336bd-7580-4b3e-9850-13ade5130ce1"). InnerVolumeSpecName "kube-api-access-sfq6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:40.896682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.896673    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca" pod "fd9336bd-7580-4b3e-9850-13ade5130ce1" (UID: "fd9336bd-7580-4b3e-9850-13ade5130ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:17:40.996397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.996323    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfq6x\" (UniqueName: \"kubernetes.io/projected/fd9336bd-7580-4b3e-9850-13ade5130ce1-kube-api-access-sfq6x\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:40.996397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:40.996352    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd9336bd-7580-4b3e-9850-13ade5130ce1-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:41.530795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.530746    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:41.531182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.531030    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:41.531833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.531813    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:17:41.598624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.598594    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"
Nov 01 20:17:41.598861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.598656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl" event={"ID":"fd9336bd-7580-4b3e-9850-13ade5130ce1","Type":"ContainerDied","Data":"339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5"}
Nov 01 20:17:41.598861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:41.598699    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339bc6c066fd63d6b6fa308664675c19ae9887e7cc0dca653b43d042d66eb4b5"
Nov 01 20:17:43.896850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:43.896820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:43.914876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:43.914848    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:44.730447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:44.730411    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:17:51.594803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.570252    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:17:51.659741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.659704    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fd9336bd-7580-4b3e-9850-13ade5130ce1" containerName="revoke-keys"
Nov 01 20:17:51.659980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.659962    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9336bd-7580-4b3e-9850-13ade5130ce1" containerName="revoke-keys"
Nov 01 20:17:51.720903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.720856    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9336bd-7580-4b3e-9850-13ade5130ce1" containerName="revoke-keys"
Nov 01 20:17:51.734362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.734315    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.742608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.742578    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"routing-overlay\""
Nov 01 20:17:51.743001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.742980    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-e2e--900891f9-dockercfg-ns4ld\""
Nov 01 20:17:51.743226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.743211    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-646b8d4cafa4\""
Nov 01 20:17:51.743437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.743422    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-85391858a611\""
Nov 01 20:17:51.743704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.743686    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bccce1\""
Nov 01 20:17:51.743950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.743933    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"openshift-service-ca.crt\""
Nov 01 20:17:51.744172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.744158    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"kube-root-ca.crt\""
Nov 01 20:17:51.780533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.768619    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:17:51.802108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.802070    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8hk\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.807935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.807906    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.808325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.808304    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.808762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.808741    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.832840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.817486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"]
Nov 01 20:17:51.869717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.869635    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"]
Nov 01 20:17:51.869914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.869749    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:51.875155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.873477    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:17:51.875372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.875353    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:17:51.875451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.875436    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:17:51.875508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.875470    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910342    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910419    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910479    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4tm\" (UniqueName: \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910532    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910620    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8hk\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910651    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.910799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.910740    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.919831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.911940    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.919831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.912557    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.919831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.914267    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:51.920430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:51.920402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8hk\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:52.011803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.011753    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.011990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.011824    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.011990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.011868    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4tm\" (UniqueName: \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.012691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.012660    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.014320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.014294    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.022005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.021978    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4tm\" (UniqueName: \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\") pod \"maas-api-revoke-sub-a708d74a-7vpml\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.167251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.167176    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:52.194073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.194042    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:52.472334 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:17:52.472254    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea27377_e6d8_488f_b4bd_361dff8edf24.slice/crio-4667af6c2f7aacd93d67826f666326dbb7aeaeac880e8ea20b37d048c8eacfcb WatchSource:0}: Error finding container 4667af6c2f7aacd93d67826f666326dbb7aeaeac880e8ea20b37d048c8eacfcb: Status 404 returned error can't find the container with id 4667af6c2f7aacd93d67826f666326dbb7aeaeac880e8ea20b37d048c8eacfcb
Nov 01 20:17:52.472537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.472398    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:17:52.494813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.494771    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"]
Nov 01 20:17:52.494896 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:17:52.494844    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a8c445_40f9_48b6_9c74_090d20611d35.slice/crio-505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd WatchSource:0}: Error finding container 505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd: Status 404 returned error can't find the container with id 505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd
Nov 01 20:17:52.861373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.861337    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml" event={"ID":"91a8c445-40f9-48b6-9c74-090d20611d35","Type":"ContainerStarted","Data":"5a953db70d0bac243be1040f99a8a241157cf2c0792330eb3509ff131d6ecbb8"}
Nov 01 20:17:52.861857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.861400    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml" event={"ID":"91a8c445-40f9-48b6-9c74-090d20611d35","Type":"ContainerStarted","Data":"505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd"}
Nov 01 20:17:52.863263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.863238    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" event={"ID":"9ea27377-e6d8-488f-b4bd-361dff8edf24","Type":"ContainerStarted","Data":"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"}
Nov 01 20:17:52.863374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.863285    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:17:52.863374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.863301    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" event={"ID":"9ea27377-e6d8-488f-b4bd-361dff8edf24","Type":"ContainerStarted","Data":"4667af6c2f7aacd93d67826f666326dbb7aeaeac880e8ea20b37d048c8eacfcb"}
Nov 01 20:17:52.877456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.877415    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml" podStartSLOduration=1.877402482 podStartE2EDuration="1.877402482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:52.87675732 +0000 UTC m=+1160.597835996" watchObservedRunningTime="2026-11-01 20:17:52.877402482 +0000 UTC m=+1160.598481158"
Nov 01 20:17:52.897540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:52.897486    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" podStartSLOduration=1.897469767 podStartE2EDuration="1.897469767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:52.892305526 +0000 UTC m=+1160.613384205" watchObservedRunningTime="2026-11-01 20:17:52.897469767 +0000 UTC m=+1160.618548447"
Nov 01 20:17:53.873418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.873387    5041 generic.go:356] "Generic (PLEG): container finished" podID="91a8c445-40f9-48b6-9c74-090d20611d35" containerID="5a953db70d0bac243be1040f99a8a241157cf2c0792330eb3509ff131d6ecbb8" exitCode=0
Nov 01 20:17:53.873914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.873469    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml" event={"ID":"91a8c445-40f9-48b6-9c74-090d20611d35","Type":"ContainerDied","Data":"5a953db70d0bac243be1040f99a8a241157cf2c0792330eb3509ff131d6ecbb8"}
Nov 01 20:17:53.875340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.875317    5041 generic.go:356] "Generic (PLEG): container finished" podID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerID="060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666" exitCode=0
Nov 01 20:17:53.875479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.875385    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerDied","Data":"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"}
Nov 01 20:17:53.875917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.875892    5041 scope.go:122] "RemoveContainer" containerID="060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"
Nov 01 20:17:53.877830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.877768    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:53.877936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.877880    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:53.878002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:53.877947    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:17:54.888482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:54.888441    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerStarted","Data":"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"}
Nov 01 20:17:55.081793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.081742    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"]
Nov 01 20:17:55.086081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.086047    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83044d3b-m9jpr"]
Nov 01 20:17:55.136655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.136629    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:55.181071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.180985    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\" (UniqueName: \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\") pod \"91a8c445-40f9-48b6-9c74-090d20611d35\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") "
Nov 01 20:17:55.181239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.181071    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\") pod \"91a8c445-40f9-48b6-9c74-090d20611d35\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") "
Nov 01 20:17:55.181239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.181134    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\") pod \"91a8c445-40f9-48b6-9c74-090d20611d35\" (UID: \"91a8c445-40f9-48b6-9c74-090d20611d35\") "
Nov 01 20:17:55.181865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.181829    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca" pod "91a8c445-40f9-48b6-9c74-090d20611d35" (UID: "91a8c445-40f9-48b6-9c74-090d20611d35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:17:55.182278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.182260    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:55.183601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.183578    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm" pod "91a8c445-40f9-48b6-9c74-090d20611d35" (UID: "91a8c445-40f9-48b6-9c74-090d20611d35"). InnerVolumeSpecName "kube-api-access-pf4tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:17:55.184029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.184009    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca" pod "91a8c445-40f9-48b6-9c74-090d20611d35" (UID: "91a8c445-40f9-48b6-9c74-090d20611d35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:17:55.229879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.229850    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:17:55.230131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.230105    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:17:55.230225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.230199    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 20:17:55.283584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.283544    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4tm\" (UniqueName: \"kubernetes.io/projected/91a8c445-40f9-48b6-9c74-090d20611d35-kube-api-access-pf4tm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:55.283584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.283580    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91a8c445-40f9-48b6-9c74-090d20611d35-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:17:55.899084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.899054    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"
Nov 01 20:17:55.899520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.899082    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml" event={"ID":"91a8c445-40f9-48b6-9c74-090d20611d35","Type":"ContainerDied","Data":"505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd"}
Nov 01 20:17:55.899520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:55.899130    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505fb0e1b6e5e4d2b040d1a188548a525dc05c3ff74b380a013a1099603821cd"
Nov 01 20:17:56.230216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:56.230134    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:17:56.230216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:56.230200    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 20:17:56.430319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:56.430281    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc87e8b-9c71-45a8-9daf-2d25ccaf3985" path="/var/lib/kubelet/pods/afc87e8b-9c71-45a8-9daf-2d25ccaf3985/volumes"
Nov 01 20:17:57.230386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:57.230354    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:17:57.230862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:57.230416    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 20:17:58.230364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:58.230332    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:17:58.230543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:58.230383    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 20:17:58.571208 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:17:58.571173    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:17:59.229585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:59.229541    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 01 20:17:59.229807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:17:59.229613    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 01 20:18:00.367647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:00.367613    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:18:00.369104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:00.369081    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:18:00.369948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:00.369924    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:18:02.243709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.243672    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"]
Nov 01 20:18:02.244645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.244622    5041 cpu_manager.go:414] "containerMap: removing container" podUID="91a8c445-40f9-48b6-9c74-090d20611d35" containerName="revoke-keys"
Nov 01 20:18:02.244707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.244648    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a8c445-40f9-48b6-9c74-090d20611d35" containerName="revoke-keys"
Nov 01 20:18:02.244840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.244826    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a8c445-40f9-48b6-9c74-090d20611d35" containerName="revoke-keys"
Nov 01 20:18:02.249761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.249735    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.254012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.253989    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:18:02.254240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.254223    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:18:02.254346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.254329    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:18:02.254418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.254407    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:18:02.255149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.255117    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"]
Nov 01 20:18:02.319125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.319092    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.319335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.319311    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jkmp\" (UniqueName: \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.319466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.319444    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.421163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.421125    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkmp\" (UniqueName: \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.421383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.421234    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.421482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.421409    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.422341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.422314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.424432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.424412    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.429293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.429269    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkmp\" (UniqueName: \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\") pod \"maas-api-revoke-sub-a741fe3e-g2rkx\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.570533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.570494    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:02.853570 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:18:02.853534    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0c57a6_4c28_495e_b871_86ba2622683a.slice/crio-62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da WatchSource:0}: Error finding container 62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da: Status 404 returned error can't find the container with id 62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da
Nov 01 20:18:02.854166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.854137    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"]
Nov 01 20:18:02.979996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:02.979958    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx" event={"ID":"3f0c57a6-4c28-495e-b871-86ba2622683a","Type":"ContainerStarted","Data":"62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da"}
Nov 01 20:18:03.878678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:03.878649    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:18:03.991196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:03.991165    5041 generic.go:356] "Generic (PLEG): container finished" podID="3f0c57a6-4c28-495e-b871-86ba2622683a" containerID="4a37290c78a0aff0b7f33b01b8fc3e8e5d4c8e25541e915f79195b569b2461eb" exitCode=0
Nov 01 20:18:03.991339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:03.991256    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx" event={"ID":"3f0c57a6-4c28-495e-b871-86ba2622683a","Type":"ContainerDied","Data":"4a37290c78a0aff0b7f33b01b8fc3e8e5d4c8e25541e915f79195b569b2461eb"}
Nov 01 20:18:05.230446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.230415    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:05.395150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.395073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\" (UniqueName: \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\") pod \"3f0c57a6-4c28-495e-b871-86ba2622683a\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") "
Nov 01 20:18:05.395150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.395134    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\") pod \"3f0c57a6-4c28-495e-b871-86ba2622683a\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") "
Nov 01 20:18:05.395360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.395274    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\") pod \"3f0c57a6-4c28-495e-b871-86ba2622683a\" (UID: \"3f0c57a6-4c28-495e-b871-86ba2622683a\") "
Nov 01 20:18:05.395636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.395608    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca" pod "3f0c57a6-4c28-495e-b871-86ba2622683a" (UID: "3f0c57a6-4c28-495e-b871-86ba2622683a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:05.395932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.395913    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:05.397892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.397865    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca" pod "3f0c57a6-4c28-495e-b871-86ba2622683a" (UID: "3f0c57a6-4c28-495e-b871-86ba2622683a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:05.397994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.397864    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp" pod "3f0c57a6-4c28-495e-b871-86ba2622683a" (UID: "3f0c57a6-4c28-495e-b871-86ba2622683a"). InnerVolumeSpecName "kube-api-access-7jkmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:05.497251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.497212    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jkmp\" (UniqueName: \"kubernetes.io/projected/3f0c57a6-4c28-495e-b871-86ba2622683a-kube-api-access-7jkmp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:05.497251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:05.497253    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f0c57a6-4c28-495e-b871-86ba2622683a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:06.063984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:06.063955    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"
Nov 01 20:18:06.064159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:06.063980    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx" event={"ID":"3f0c57a6-4c28-495e-b871-86ba2622683a","Type":"ContainerDied","Data":"62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da"}
Nov 01 20:18:06.064159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:06.064027    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a2be7dda6dd5154e21d575d9ed0caea4e2282764a32a4802e11ecf2db773da"
Nov 01 20:18:14.182022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.181985    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"]
Nov 01 20:18:14.202739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.202706    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3f0c57a6-4c28-495e-b871-86ba2622683a" containerName="revoke-keys"
Nov 01 20:18:14.202739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.202735    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0c57a6-4c28-495e-b871-86ba2622683a" containerName="revoke-keys"
Nov 01 20:18:14.206109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.206009    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0c57a6-4c28-495e-b871-86ba2622683a" containerName="revoke-keys"
Nov 01 20:18:14.213092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.213066    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.215653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.214968    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"]
Nov 01 20:18:14.223422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.223084    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:18:14.223422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.223337    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:18:14.223885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.223835    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:18:14.225331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.224659    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:18:14.405530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.405487    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmds\" (UniqueName: \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.405790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.405756    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.406018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.405993    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.507607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.507504    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.507881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.507645    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmds\" (UniqueName: \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.510076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.508139    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.510076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.508958    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.512226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.512189    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.516285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.516256    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmds\" (UniqueName: \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\") pod \"maas-api-revoke-sub-059ea56a-4vzcv\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.533209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.533181    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:14.818601 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:18:14.818571    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf85b3_9d21_4a63_ae3c_adc8c14f3ee7.slice/crio-4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988 WatchSource:0}: Error finding container 4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988: Status 404 returned error can't find the container with id 4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988
Nov 01 20:18:14.818852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:14.818825    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"]
Nov 01 20:18:15.243862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:15.243696    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv" event={"ID":"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7","Type":"ContainerStarted","Data":"4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988"}
Nov 01 20:18:16.255596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:16.255561    5041 generic.go:356] "Generic (PLEG): container finished" podID="f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" containerID="56deef347f66390b256bf195d578cb5418446716ff01240da37c5e40b8f6cacf" exitCode=0
Nov 01 20:18:16.256071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:16.255638    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv" event={"ID":"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7","Type":"ContainerDied","Data":"56deef347f66390b256bf195d578cb5418446716ff01240da37c5e40b8f6cacf"}
Nov 01 20:18:17.441278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.441249    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:17.452139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.452107    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\") pod \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") "
Nov 01 20:18:17.452277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.452156    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\") pod \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") "
Nov 01 20:18:17.452322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.452304    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\" (UniqueName: \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\") pod \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\" (UID: \"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7\") "
Nov 01 20:18:17.452595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.452568    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca" pod "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" (UID: "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:17.452743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.452728    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:17.454191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.454165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca" pod "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" (UID: "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:17.454376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.454356    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds" pod "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" (UID: "f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7"). InnerVolumeSpecName "kube-api-access-nbmds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:17.553882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.553847    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmds\" (UniqueName: \"kubernetes.io/projected/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-kube-api-access-nbmds\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:17.553882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:17.553875    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:18.277617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:18.277588    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"
Nov 01 20:18:18.277617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:18.277613    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv" event={"ID":"f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7","Type":"ContainerDied","Data":"4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988"}
Nov 01 20:18:18.277845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:18.277657    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6f5b72511af57b98dfb02fdc335dd080d43e07922017a7f87608cdc51b8988"
Nov 01 20:18:24.725741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.725709    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"]
Nov 01 20:18:24.726430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.726416    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" containerName="revoke-keys"
Nov 01 20:18:24.726475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.726432    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" containerName="revoke-keys"
Nov 01 20:18:24.726593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.726582    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" containerName="revoke-keys"
Nov 01 20:18:24.730410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.730395    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.733848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.733825    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:18:24.734013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.733865    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:18:24.734958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.734936    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:18:24.735119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.734958    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:18:24.736764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.736743    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"]
Nov 01 20:18:24.863395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.863359    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvj6\" (UniqueName: \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.863585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.863510    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.863648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.863626    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.965990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.965954    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.966214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.966087    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.966290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.966213    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvj6\" (UniqueName: \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.967059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.967034    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.968551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.968530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:24.974119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:24.974089    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvj6\" (UniqueName: \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\") pod \"maas-api-revoke-sub-d9ad4fad-n69n4\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:25.048307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:25.048273    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:25.335361 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:18:25.335327    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ccfed8_4f8f_412b_9ef8_69ce0efe71f8.slice/crio-aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d WatchSource:0}: Error finding container aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d: Status 404 returned error can't find the container with id aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d
Nov 01 20:18:25.335687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:25.335664    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"]
Nov 01 20:18:25.671270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:25.671235    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4" event={"ID":"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8","Type":"ContainerStarted","Data":"3dfb5991b6e104f2ef6bbe9e0ef88f773abc7dc0f61556756f7bcbadcf7c5566"}
Nov 01 20:18:25.671485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:25.671302    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4" event={"ID":"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8","Type":"ContainerStarted","Data":"aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d"}
Nov 01 20:18:25.691485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:25.691434    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4" podStartSLOduration=1.691419497 podStartE2EDuration="1.691419497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:25.688285627 +0000 UTC m=+1193.409364304" watchObservedRunningTime="2026-11-01 20:18:25.691419497 +0000 UTC m=+1193.412498173"
Nov 01 20:18:26.681580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.681546    5041 generic.go:356] "Generic (PLEG): container finished" podID="a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" containerID="3dfb5991b6e104f2ef6bbe9e0ef88f773abc7dc0f61556756f7bcbadcf7c5566" exitCode=0
Nov 01 20:18:26.681985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.681626    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4" event={"ID":"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8","Type":"ContainerDied","Data":"3dfb5991b6e104f2ef6bbe9e0ef88f773abc7dc0f61556756f7bcbadcf7c5566"}
Nov 01 20:18:26.858817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.858787    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-c5hj4-6c5dr"]
Nov 01 20:18:26.863633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.863612    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:26.871147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.870759    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c5hj4-6c5dr"]
Nov 01 20:18:26.926066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.926025    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:18:26.926360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.926321    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy" containerID="cri-o://8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1" gracePeriod=30
Nov 01 20:18:26.929539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.929475    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-96bd4\""
Nov 01 20:18:26.964384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.964297    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:18:26.993825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.993769    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:26.994017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.993927    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srspp\" (UniqueName: \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:26.994017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.993961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:26.994201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:26.994129    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095215    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095319    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srspp\" (UniqueName: \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095407    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095757    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.095959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.095939    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.096138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.096121    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.103104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.103078    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srspp\" (UniqueName: \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\") pod \"periodic-gathering-c5hj4-6c5dr\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") " pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.190821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.189035    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:18:27.504680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.504647    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-c5hj4-6c5dr"]
Nov 01 20:18:27.693132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.693091    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr" event={"ID":"d60f81ca-b7da-4b99-899c-f8ba5651ffa2","Type":"ContainerStarted","Data":"7201b96ecd21d468b4568bd957fb65105834d2a831f3736fa70ddb52d8a053a6"}
Nov 01 20:18:27.693556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.693153    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr" event={"ID":"d60f81ca-b7da-4b99-899c-f8ba5651ffa2","Type":"ContainerStarted","Data":"4edcbc87fb677bcc5dab7a78f7515c15ddd11fe4b94059db0c7ee9e9487ee4b7"}
Nov 01 20:18:27.710957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.710899    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr" podStartSLOduration=1.710880677 podStartE2EDuration="1.710880677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:27.708421807 +0000 UTC m=+1195.429500484" watchObservedRunningTime="2026-11-01 20:18:27.710880677 +0000 UTC m=+1195.431959355"
Nov 01 20:18:27.859322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.859286    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:27.919547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.919516    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\") pod \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") "
Nov 01 20:18:27.919745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.919674    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\" (UniqueName: \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\") pod \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") "
Nov 01 20:18:27.919745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.919706    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\") pod \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\" (UID: \"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8\") "
Nov 01 20:18:27.920196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.920167    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca" pod "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" (UID: "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:27.920305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.920290    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:27.921659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.921640    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca" pod "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" (UID: "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:18:27.921950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:27.921920    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6" pod "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" (UID: "a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8"). InnerVolumeSpecName "kube-api-access-6cvj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:28.022373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:28.022290    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvj6\" (UniqueName: \"kubernetes.io/projected/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-kube-api-access-6cvj6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:28.022373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:28.022319    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:28.542837 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:18:28.542790    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:18:28.709354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:28.703174    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"
Nov 01 20:18:28.709354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:28.703211    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4" event={"ID":"a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8","Type":"ContainerDied","Data":"aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d"}
Nov 01 20:18:28.709354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:28.703288    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabca97aa0ab5707f865e92c67c7e9bec562cb80cfbf171dc49560bf73c6192d"
Nov 01 20:18:31.964863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:31.964829    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" podUID="9ea27377-e6d8-488f-b4bd-361dff8edf24" containerName="payload-processing" containerID="cri-o://dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634" gracePeriod=25
Nov 01 20:18:32.278305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.278272    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:18:32.294072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.293979    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:18:32.388234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\") pod \"9ea27377-e6d8-488f-b4bd-361dff8edf24\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") "
Nov 01 20:18:32.388234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388252    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388371    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\") pod \"9ea27377-e6d8-488f-b4bd-361dff8edf24\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") "
Nov 01 20:18:32.388530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388413    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388448    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388480    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388521    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388576    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\") pod \"9ea27377-e6d8-488f-b4bd-361dff8edf24\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") "
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388607    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388670    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388721    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388741    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume" pod "9ea27377-e6d8-488f-b4bd-361dff8edf24" (UID: "9ea27377-e6d8-488f-b4bd-361dff8edf24"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:32.388838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388756    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\") pod \"9ea27377-e6d8-488f-b4bd-361dff8edf24\" (UID: \"9ea27377-e6d8-488f-b4bd-361dff8edf24\") "
Nov 01 20:18:32.389176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388853    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:32.389176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388873    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:32.389176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388907    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.389176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388942    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume" pod "9ea27377-e6d8-488f-b4bd-361dff8edf24" (UID: "9ea27377-e6d8-488f-b4bd-361dff8edf24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:32.389176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.388946    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\") pod \"68b4d85e-e38b-4544-a97a-52dd68367199\" (UID: \"68b4d85e-e38b-4544-a97a-52dd68367199\") "
Nov 01 20:18:32.389481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.389276    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:32.389824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.389746    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:18:32.390387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390364    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b4d85e-e38b-4544-a97a-52dd68367199-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390391    5041 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-routing-overlay-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390406    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390425    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea27377-e6d8-488f-b4bd-361dff8edf24-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390440    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390453    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.390493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.390467    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.391252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391227    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:32.391373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391342    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials" pod "9ea27377-e6d8-488f-b4bd-361dff8edf24" (UID: "9ea27377-e6d8-488f-b4bd-361dff8edf24"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:32.391453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391434    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk" pod "9ea27377-e6d8-488f-b4bd-361dff8edf24" (UID: "9ea27377-e6d8-488f-b4bd-361dff8edf24"). InnerVolumeSpecName "kube-api-access-9l8hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:32.391497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391463    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "kube-api-access-mkv2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:18:32.391620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391599    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:18:32.391827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.391804    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy" pod "68b4d85e-e38b-4544-a97a-52dd68367199" (UID: "68b4d85e-e38b-4544-a97a-52dd68367199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:18:32.491703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491621    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b4d85e-e38b-4544-a97a-52dd68367199-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.491703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491655    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkv2z\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-kube-api-access-mkv2z\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.491703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491671    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8hk\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-kube-api-access-9l8hk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.491703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491685    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b4d85e-e38b-4544-a97a-52dd68367199-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.491703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491698    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b4d85e-e38b-4544-a97a-52dd68367199-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.492014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.491711    5041 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ea27377-e6d8-488f-b4bd-361dff8edf24-provider-credentials\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:18:32.700618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.700583    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:18:32.704828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.704803    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:18:32.736335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.736305    5041 generic.go:356] "Generic (PLEG): container finished" podID="68b4d85e-e38b-4544-a97a-52dd68367199" containerID="8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1" exitCode=0
Nov 01 20:18:32.736517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.736378    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"
Nov 01 20:18:32.736517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.736389    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" event={"ID":"68b4d85e-e38b-4544-a97a-52dd68367199","Type":"ContainerDied","Data":"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"}
Nov 01 20:18:32.736517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.736473    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5" event={"ID":"68b4d85e-e38b-4544-a97a-52dd68367199","Type":"ContainerDied","Data":"29f063176574f07aa8dd7d61c61bd5a8fa3651ff0ee5d9abdbf29c0b6ef2bbba"}
Nov 01 20:18:32.736517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.736498    5041 scope.go:122] "RemoveContainer" containerID="8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"
Nov 01 20:18:32.738233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.738205    5041 generic.go:356] "Generic (PLEG): container finished" podID="9ea27377-e6d8-488f-b4bd-361dff8edf24" containerID="dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634" exitCode=0
Nov 01 20:18:32.738350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.738284    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"
Nov 01 20:18:32.738350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.738337    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" event={"ID":"9ea27377-e6d8-488f-b4bd-361dff8edf24","Type":"ContainerDied","Data":"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"}
Nov 01 20:18:32.738437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.738378    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd" event={"ID":"9ea27377-e6d8-488f-b4bd-361dff8edf24","Type":"ContainerDied","Data":"4667af6c2f7aacd93d67826f666326dbb7aeaeac880e8ea20b37d048c8eacfcb"}
Nov 01 20:18:32.748949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.748926    5041 scope.go:122] "RemoveContainer" containerID="8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"
Nov 01 20:18:32.749218 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:18:32.749201    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1\": container with ID starting with 8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1 not found: ID does not exist" containerID="8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"
Nov 01 20:18:32.749275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.749226    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1"} err="failed to get container status \"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1\": rpc error: code = NotFound desc = could not find container \"8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1\": container with ID starting with 8ae7ed2965a0283c5241771bf7f1aff50b2a2737c7332aeba11d66d5764f6af1 not found: ID does not exist"
Nov 01 20:18:32.749275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.749242    5041 scope.go:122] "RemoveContainer" containerID="dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"
Nov 01 20:18:32.759047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.759025    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:18:32.762208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.762189    5041 scope.go:122] "RemoveContainer" containerID="dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"
Nov 01 20:18:32.762502 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:18:32.762483    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634\": container with ID starting with dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634 not found: ID does not exist" containerID="dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"
Nov 01 20:18:32.762571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.762511    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634"} err="failed to get container status \"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634\": rpc error: code = NotFound desc = could not find container \"dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634\": container with ID starting with dc4129b1f42102e09da809208787804ca707dccae65222d2c54931485c18a634 not found: ID does not exist"
Nov 01 20:18:32.763598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.763579    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-dwds5"]
Nov 01 20:18:32.778225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.778197    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:18:32.782160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:32.782135    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-679d4gzpd"]
Nov 01 20:18:34.430188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.430151    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" path="/var/lib/kubelet/pods/68b4d85e-e38b-4544-a97a-52dd68367199/volumes"
Nov 01 20:18:34.430771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.430741    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea27377-e6d8-488f-b4bd-361dff8edf24" path="/var/lib/kubelet/pods/9ea27377-e6d8-488f-b4bd-361dff8edf24/volumes"
Nov 01 20:18:34.948508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.948474    5041 scope.go:122] "RemoveContainer" containerID="f8862d0bed1b176565eaab662c0bdaa0104381b342c1e042cbc2e95a7b4ae13b"
Nov 01 20:18:34.958857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.958834    5041 scope.go:122] "RemoveContainer" containerID="0044fdc3b1aee578105c75634c09e4d0e92e41abd0c78572f15e39c32398f090"
Nov 01 20:18:34.969165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.969144    5041 scope.go:122] "RemoveContainer" containerID="03df1ae0237ec4b749e360eb1c3bdbfc8eb261fb6fcd282b514201aef6cba0f8"
Nov 01 20:18:34.978665 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.978644    5041 scope.go:122] "RemoveContainer" containerID="64dd1d761ad5e322fb48131d1e58599874b6b4a47001089c5cbc4a6464815259"
Nov 01 20:18:34.988251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.988231    5041 scope.go:122] "RemoveContainer" containerID="1dbeaa47bf274c34f6f42de8b57ddd3bb8d53ecc103b597aa94ebdb1c858c7b5"
Nov 01 20:18:34.998066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:34.998044    5041 scope.go:122] "RemoveContainer" containerID="718c196a8bcce3812e24962d31b87053469295acb35315136e596845b49ce005"
Nov 01 20:18:35.012175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:35.012147    5041 scope.go:122] "RemoveContainer" containerID="76df7018dfe1c1374a3d43d8b726e8cda01f81aba37318e941a9b42bb970cbac"
Nov 01 20:18:42.296803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:42.294120    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:18:43.813833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.813690    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:18:43.814814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814743    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" containerName="revoke-keys"
Nov 01 20:18:43.814814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814773    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" containerName="revoke-keys"
Nov 01 20:18:43.815013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814834    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9ea27377-e6d8-488f-b4bd-361dff8edf24" containerName="payload-processing"
Nov 01 20:18:43.815013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814844    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea27377-e6d8-488f-b4bd-361dff8edf24" containerName="payload-processing"
Nov 01 20:18:43.815013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814861    5041 cpu_manager.go:414] "containerMap: removing container" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy"
Nov 01 20:18:43.815013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.814869    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy"
Nov 01 20:18:43.815265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.815037    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea27377-e6d8-488f-b4bd-361dff8edf24" containerName="payload-processing"
Nov 01 20:18:43.815265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.815054    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b4d85e-e38b-4544-a97a-52dd68367199" containerName="istio-proxy"
Nov 01 20:18:43.815265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.815072    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" containerName="revoke-keys"
Nov 01 20:18:43.847295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.843022    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:18:43.847295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.843136    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.847295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.847043    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3de00d-openshift-default-dockercfg-gc9qb\""
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.923956    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924007    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96fv\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924059    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924101    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924135    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924181    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924238    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924538    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:43.925798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:43.924613    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.032940    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033079    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033104    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96fv\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033145    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033216    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033258    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.033804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.033312    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.034446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.034265    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.044984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.038861    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.044984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.039316    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.044984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.039935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.044984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.040168    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.045734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.045658    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.082481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.082403    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.139808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.082537    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96fv\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.191456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.191418    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\") pod \"e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") " pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.215800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.210521    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:44.555933 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:18:44.555880    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108e9100_17c0_473e_9605_800b410f5d61.slice/crio-a954c400842806cebfded060467b2e8bc845c59b73d63545522d0c587b7c3fc9 WatchSource:0}: Error finding container a954c400842806cebfded060467b2e8bc845c59b73d63545522d0c587b7c3fc9: Status 404 returned error can't find the container with id a954c400842806cebfded060467b2e8bc845c59b73d63545522d0c587b7c3fc9
Nov 01 20:18:44.556654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.556548    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:18:44.559364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.559322    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:18:44.559473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.559415    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:18:44.559473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:44.559460    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:18:45.131159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:45.131119    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerStarted","Data":"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"}
Nov 01 20:18:45.131589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:45.131179    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerStarted","Data":"a954c400842806cebfded060467b2e8bc845c59b73d63545522d0c587b7c3fc9"}
Nov 01 20:18:45.151189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:45.151140    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podStartSLOduration=2.151127378 podStartE2EDuration="2.151127378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:45.149118541 +0000 UTC m=+1212.870197218" watchObservedRunningTime="2026-11-01 20:18:45.151127378 +0000 UTC m=+1212.872206115"
Nov 01 20:18:45.211412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:45.211378    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:45.217092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:45.217066    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:46.139516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:46.139483    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:46.140550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:46.140522    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:18:54.077865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:54.077831    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"]
Nov 01 20:18:54.082203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:54.082169    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bebfd38-zg6jc"]
Nov 01 20:18:54.446863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:54.445525    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd07240-059e-4623-b259-5eb0efe8e43f" path="/var/lib/kubelet/pods/8bd07240-059e-4623-b259-5eb0efe8e43f/volumes"
Nov 01 20:18:57.053803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:57.050548    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:18:57.261802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:57.258869    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:18:57.261802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:57.259178    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:18:57.305857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:57.300963    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:18:57.864804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:18:57.864465    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:18:58.542915 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:18:58.542866    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:19:00.974551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:00.974517    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:19:00.986314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:00.986284    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.009450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.009264    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-3f909928dd00\""
Nov 01 20:19:01.009450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.009415    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\""
Nov 01 20:19:01.010206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.010000    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\""
Nov 01 20:19:01.010206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.010109    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\""
Nov 01 20:19:01.010206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.010179    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\""
Nov 01 20:19:01.010501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.010472    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:19:01.027720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.027249    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:19:01.079492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.079450    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjjr\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.079720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.079609    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.079720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.079718    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.079931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.079748    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.093258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.093099    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\""
Nov 01 20:19:01.093258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.093180    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-85078fdb41c3\""
Nov 01 20:19:01.093258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.093229    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-8876b9054304\""
Nov 01 20:19:01.194228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.192216    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.194228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.192297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.194228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.192450    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjjr\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.194228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.192609    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.194228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.193769    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.197893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.197152    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.200450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.198101    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.205988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.205966    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjjr\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.352226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.352193    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:01.764099 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:19:01.754355    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef565c9_e4af_46b3_ab54_66af348e0872.slice/crio-6bfedfaa3419a67f222e97329cec4e5bddfa612301ea6208a79a621c5aa4ba0f WatchSource:0}: Error finding container 6bfedfaa3419a67f222e97329cec4e5bddfa612301ea6208a79a621c5aa4ba0f: Status 404 returned error can't find the container with id 6bfedfaa3419a67f222e97329cec4e5bddfa612301ea6208a79a621c5aa4ba0f
Nov 01 20:19:01.764099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:01.761973    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:19:02.663756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:02.663719    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" event={"ID":"1ef565c9-e4af-46b3-ab54-66af348e0872","Type":"ContainerStarted","Data":"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"}
Nov 01 20:19:02.664229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:02.663855    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:02.664229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:02.663879    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" event={"ID":"1ef565c9-e4af-46b3-ab54-66af348e0872","Type":"ContainerStarted","Data":"6bfedfaa3419a67f222e97329cec4e5bddfa612301ea6208a79a621c5aa4ba0f"}
Nov 01 20:19:02.684028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:02.683974    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" podStartSLOduration=2.683961058 podStartE2EDuration="2.683961058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:02.679739048 +0000 UTC m=+1230.400817738" watchObservedRunningTime="2026-11-01 20:19:02.683961058 +0000 UTC m=+1230.405039734"
Nov 01 20:19:13.463067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.463033    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:19:13.470611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.470587    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.490846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.490813    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6afe1a-openshift-default-dockercfg-dcbn5\""
Nov 01 20:19:13.500223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.500197    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:19:13.533711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533672    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.533930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533718    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.533930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533746    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.533930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533791    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.534107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.534107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.533980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.534211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.534191    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq54j\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.534267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.534231    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.534325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.534289    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635237    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635255    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635278    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635387    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635411    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635524    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq54j\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635555    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635604    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635644    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635728    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.635928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635748    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.636007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.635962    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.636072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.636049    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.637865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.637841    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.638140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.638119    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.643346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.643326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.643559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.643541    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq54j\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\") pod \"e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") " pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:13.735308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.734000    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:19:13.839874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:13.814008    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:14.234029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.231110    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:19:14.234029 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:19:14.233990    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525af09b_2ace_4907_b979_2e9439ce47a0.slice/crio-83dff9ed927dfb5525059326c02bad8dd8cf2d60aecf443057de768cd9ad86d7 WatchSource:0}: Error finding container 83dff9ed927dfb5525059326c02bad8dd8cf2d60aecf443057de768cd9ad86d7: Status 404 returned error can't find the container with id 83dff9ed927dfb5525059326c02bad8dd8cf2d60aecf443057de768cd9ad86d7
Nov 01 20:19:14.237233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.236947    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:14.237233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.237052    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:14.237233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.237119    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:14.926860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.926820    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerStarted","Data":"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"}
Nov 01 20:19:14.927284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.926891    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerStarted","Data":"83dff9ed927dfb5525059326c02bad8dd8cf2d60aecf443057de768cd9ad86d7"}
Nov 01 20:19:14.945642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:14.945593    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podStartSLOduration=1.9455807680000001 podStartE2EDuration="1.945580768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:14.944853037 +0000 UTC m=+1242.665931713" watchObservedRunningTime="2026-11-01 20:19:14.945580768 +0000 UTC m=+1242.666659444"
Nov 01 20:19:15.841323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:15.841289    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:15.847636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:15.847611    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:15.934275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:15.934245    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:15.935221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:15.935202    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:19.740819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:19.728878    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:20.058858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:20.050005    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:20.277925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:20.277880    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:20.339437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:20.339332    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:20.735561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:20.734879    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": context deadline exceeded"
Nov 01 20:19:20.837806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:20.784712    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:19:21.056834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.053758    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:19:21.056834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.054352    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:19:21.237027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.236968    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:21.340345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.340252    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:21.341124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.341099    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:19:21.360415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.360379    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": context deadline exceeded"
Nov 01 20:19:21.360975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.360954    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:19:21.739304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.739205    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:21.941061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:21.941031    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:19:22.014104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.014021    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:19:22.076804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.050682    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:19:22.305246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.234277    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:19:22.305246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.297145    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 01 20:19:22.305246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.298290    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:19:22.335015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:22.334872    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:19:28.542529 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:19:28.542491    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:19:29.475892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.475854    5041 generic.go:356] "Generic (PLEG): container finished" podID="108e9100-17c0-473e-9605-800b410f5d61" containerID="e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53" exitCode=0
Nov 01 20:19:29.476070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.475927    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerDied","Data":"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"}
Nov 01 20:19:29.476411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.476394    5041 scope.go:122] "RemoveContainer" containerID="e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"
Nov 01 20:19:29.478009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.477978    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:29.478084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.478052    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:29.478120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:29.478083    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:30.494858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:30.494821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerStarted","Data":"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"}
Nov 01 20:19:31.140317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:31.140283    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:19:31.211685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:31.211651    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:19:31.212502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:31.212480    5041 patch_prober.go:29] interesting pod/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 01 20:19:31.212558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:31.212540    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 01 20:19:32.211867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:32.211832    5041 patch_prober.go:29] interesting pod/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 01 20:19:32.212283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:32.211884    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 01 20:19:33.211586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:33.211550    5041 patch_prober.go:29] interesting pod/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 01 20:19:33.211761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:33.211613    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 01 20:19:34.211108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:34.211069    5041 patch_prober.go:29] interesting pod/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 01 20:19:34.211523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:34.211137    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 01 20:19:35.086682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:35.086650    5041 scope.go:122] "RemoveContainer" containerID="84f14528d067b4df69d8ad4d9ae671d0ae488f2ac736f8cbbb311d804e7202fb"
Nov 01 20:19:35.097419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:35.097395    5041 scope.go:122] "RemoveContainer" containerID="f4754e0f041ad25744796cdcfd192027caaa2b70437eade5bc5d1f7857557957"
Nov 01 20:19:35.107100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:35.107085    5041 scope.go:122] "RemoveContainer" containerID="4d377f1d92d8631f2b6df45ce05cced024e901b46efa5df8e2ece90243ad7f15"
Nov 01 20:19:35.211918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:35.211880    5041 patch_prober.go:29] interesting pod/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 01 20:19:35.212301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:35.211950    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 01 20:19:36.216580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:36.216550    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:19:36.217745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:36.217721    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:19:41.950024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:41.946628    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:42.957926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:42.957872    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:19:42.959077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:42.959051    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:19:43.424737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:43.424703    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:19:50.648110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.648073    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"]
Nov 01 20:19:50.686720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.686686    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"]
Nov 01 20:19:50.687061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.686929    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.694727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.694701    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:19:50.694918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.694874    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:19:50.695059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.695042    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:19:50.695122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.695080    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:19:50.764457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.764418    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.764647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.764627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdld\" (UniqueName: \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.764722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.764665    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.866457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.866409    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdld\" (UniqueName: \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.866637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.866498    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.866716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.866639    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.867437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.867404    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.868994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.868962    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:50.873960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:50.873938    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdld\" (UniqueName: \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\") pod \"maas-api-revoke-sub-b276f2e3-kddx4\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:51.005534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.005448    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:51.251670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.250911    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"]
Nov 01 20:19:51.300623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.300587    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"]
Nov 01 20:19:51.300846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.300717    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.349312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.349284    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"]
Nov 01 20:19:51.349543 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:19:51.349518    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb591ff86_68e6_4d25_9fa3_2baafd863077.slice/crio-3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00 WatchSource:0}: Error finding container 3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00: Status 404 returned error can't find the container with id 3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00
Nov 01 20:19:51.377125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.377101    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.377289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.377261    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.377368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.377332    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvtq\" (UniqueName: \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.478434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.478402    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.478608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.478480    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvtq\" (UniqueName: \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.478676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.478630    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.479304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.479278    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.480755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.480735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.486461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.486436    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvtq\" (UniqueName: \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\") pod \"maas-api-revoke-sub-07be0125-rq8gt\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.677242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.677212    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:51.861647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.859983    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4" event={"ID":"b591ff86-68e6-4d25-9fa3-2baafd863077","Type":"ContainerStarted","Data":"8030bbd670adb085b9a168d13eaf243bfdc7e457da20e1a870202c7fe3d2680d"}
Nov 01 20:19:51.861647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.860056    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4" event={"ID":"b591ff86-68e6-4d25-9fa3-2baafd863077","Type":"ContainerStarted","Data":"3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00"}
Nov 01 20:19:51.877900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:51.877840    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4" podStartSLOduration=1.8778193330000001 podStartE2EDuration="1.877819333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:51.876609439 +0000 UTC m=+1279.597688116" watchObservedRunningTime="2026-11-01 20:19:51.877819333 +0000 UTC m=+1279.598898010"
Nov 01 20:19:52.019688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.019654    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"]
Nov 01 20:19:52.019946 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:19:52.019920    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b4fa09_c1da_4350_9efe_e06a2dc06f84.slice/crio-a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a WatchSource:0}: Error finding container a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a: Status 404 returned error can't find the container with id a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a
Nov 01 20:19:52.873493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.873455    5041 generic.go:356] "Generic (PLEG): container finished" podID="b591ff86-68e6-4d25-9fa3-2baafd863077" containerID="8030bbd670adb085b9a168d13eaf243bfdc7e457da20e1a870202c7fe3d2680d" exitCode=0
Nov 01 20:19:52.873976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.873548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4" event={"ID":"b591ff86-68e6-4d25-9fa3-2baafd863077","Type":"ContainerDied","Data":"8030bbd670adb085b9a168d13eaf243bfdc7e457da20e1a870202c7fe3d2680d"}
Nov 01 20:19:52.876023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.875996    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt" event={"ID":"f1b4fa09-c1da-4350-9efe-e06a2dc06f84","Type":"ContainerStarted","Data":"6b831f797b866706c32dfb7a6f06d0d1cf707c2ceb1d6846697409183898e37d"}
Nov 01 20:19:52.876130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.876043    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt" event={"ID":"f1b4fa09-c1da-4350-9efe-e06a2dc06f84","Type":"ContainerStarted","Data":"a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a"}
Nov 01 20:19:52.910914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:52.910855    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt" podStartSLOduration=1.910841078 podStartE2EDuration="1.910841078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:52.906860847 +0000 UTC m=+1280.627939520" watchObservedRunningTime="2026-11-01 20:19:52.910841078 +0000 UTC m=+1280.631919751"
Nov 01 20:19:53.890518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.890484    5041 generic.go:356] "Generic (PLEG): container finished" podID="525af09b-2ace-4907-b979-2e9439ce47a0" containerID="fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40" exitCode=0
Nov 01 20:19:53.891024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.890560    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerDied","Data":"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"}
Nov 01 20:19:53.891079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.891058    5041 scope.go:122] "RemoveContainer" containerID="fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"
Nov 01 20:19:53.892397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.892373    5041 generic.go:356] "Generic (PLEG): container finished" podID="f1b4fa09-c1da-4350-9efe-e06a2dc06f84" containerID="6b831f797b866706c32dfb7a6f06d0d1cf707c2ceb1d6846697409183898e37d" exitCode=0
Nov 01 20:19:53.892524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.892459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt" event={"ID":"f1b4fa09-c1da-4350-9efe-e06a2dc06f84","Type":"ContainerDied","Data":"6b831f797b866706c32dfb7a6f06d0d1cf707c2ceb1d6846697409183898e37d"}
Nov 01 20:19:53.892874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.892817    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:53.892950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.892902    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:53.893014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:53.892951    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:19:54.145826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.145735    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:54.286830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.286792    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\") pod \"b591ff86-68e6-4d25-9fa3-2baafd863077\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") "
Nov 01 20:19:54.287005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.286937    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\" (UniqueName: \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\") pod \"b591ff86-68e6-4d25-9fa3-2baafd863077\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") "
Nov 01 20:19:54.287080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.287062    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\") pod \"b591ff86-68e6-4d25-9fa3-2baafd863077\" (UID: \"b591ff86-68e6-4d25-9fa3-2baafd863077\") "
Nov 01 20:19:54.287192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.287174    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca" pod "b591ff86-68e6-4d25-9fa3-2baafd863077" (UID: "b591ff86-68e6-4d25-9fa3-2baafd863077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:54.287561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.287539    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:54.289085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.289052    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca" pod "b591ff86-68e6-4d25-9fa3-2baafd863077" (UID: "b591ff86-68e6-4d25-9fa3-2baafd863077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:19:54.289219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.289135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld" pod "b591ff86-68e6-4d25-9fa3-2baafd863077" (UID: "b591ff86-68e6-4d25-9fa3-2baafd863077"). InnerVolumeSpecName "kube-api-access-bpdld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:54.389261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.389230    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdld\" (UniqueName: \"kubernetes.io/projected/b591ff86-68e6-4d25-9fa3-2baafd863077-kube-api-access-bpdld\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:54.389261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.389257    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b591ff86-68e6-4d25-9fa3-2baafd863077-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:54.902734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.902703    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"
Nov 01 20:19:54.902734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.902716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4" event={"ID":"b591ff86-68e6-4d25-9fa3-2baafd863077","Type":"ContainerDied","Data":"3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00"}
Nov 01 20:19:54.903256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.902808    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe060c7b55b06887e40ec27160203ff1adb776bb83762a85cb5fdab767e0a00"
Nov 01 20:19:54.905333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:54.905307    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerStarted","Data":"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"}
Nov 01 20:19:55.102508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.102485    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:55.210877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.210746    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\") pod \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") "
Nov 01 20:19:55.212832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.212798    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\") pod \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") "
Nov 01 20:19:55.212989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.212885    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\" (UniqueName: \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\") pod \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\" (UID: \"f1b4fa09-c1da-4350-9efe-e06a2dc06f84\") "
Nov 01 20:19:55.213289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.212699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca" pod "f1b4fa09-c1da-4350-9efe-e06a2dc06f84" (UID: "f1b4fa09-c1da-4350-9efe-e06a2dc06f84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:19:55.215237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.215197    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:55.217135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.217093    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq" pod "f1b4fa09-c1da-4350-9efe-e06a2dc06f84" (UID: "f1b4fa09-c1da-4350-9efe-e06a2dc06f84"). InnerVolumeSpecName "kube-api-access-8gvtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:19:55.247798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.244164    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca" pod "f1b4fa09-c1da-4350-9efe-e06a2dc06f84" (UID: "f1b4fa09-c1da-4350-9efe-e06a2dc06f84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:19:55.317028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.316994    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:55.317028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.317025    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvtq\" (UniqueName: \"kubernetes.io/projected/f1b4fa09-c1da-4350-9efe-e06a2dc06f84-kube-api-access-8gvtq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:19:55.852839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.848645    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:19:55.868670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.867043    5041 patch_prober.go:29] interesting pod/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 01 20:19:55.868670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.867116    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 01 20:19:55.949446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.949415    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"
Nov 01 20:19:56.027816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:55.951929    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt" event={"ID":"f1b4fa09-c1da-4350-9efe-e06a2dc06f84","Type":"ContainerDied","Data":"a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a"}
Nov 01 20:19:56.027816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:56.008009    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20389e3285b326543760eaa10e36798b9d639f68c4d8deb8270a66175b1104a"
Nov 01 20:19:56.840879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:56.840839    5041 patch_prober.go:29] interesting pod/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 01 20:19:56.841100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:56.840902    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 01 20:19:57.840727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:57.840682    5041 patch_prober.go:29] interesting pod/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 01 20:19:57.841363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:57.840740    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 01 20:19:58.022496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.022461    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"]
Nov 01 20:19:58.027020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.026990    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-282dc1e6-rwnp8"]
Nov 01 20:19:58.432577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.432409    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc243873-82c7-43d8-8851-1334401203b2" path="/var/lib/kubelet/pods/bc243873-82c7-43d8-8851-1334401203b2/volumes"
Nov 01 20:19:58.568799 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:19:58.552117    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:19:58.654153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.654121    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:19:58.654624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.654592    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy" containerID="cri-o://985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5" gracePeriod=30
Nov 01 20:19:58.861801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.857897    5041 patch_prober.go:29] interesting pod/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 01 20:19:58.861801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:58.857965    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 01 20:19:59.841355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:59.841319    5041 patch_prober.go:29] interesting pod/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Nov 01 20:19:59.841545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:19:59.841368    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Nov 01 20:20:00.025675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.025638    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"]
Nov 01 20:20:00.030331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.030303    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b6f9dc-8cqhh"]
Nov 01 20:20:00.434645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.434605    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83737883-64a0-4dcb-b495-6d5ffd6d0a1c" path="/var/lib/kubelet/pods/83737883-64a0-4dcb-b495-6d5ffd6d0a1c/volumes"
Nov 01 20:20:00.845837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.845773    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:20:00.846329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.846309    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:20:00.847007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.846990    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:20:00.861502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.861468    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"]
Nov 01 20:20:00.862147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862130    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b591ff86-68e6-4d25-9fa3-2baafd863077" containerName="revoke-keys"
Nov 01 20:20:00.862147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862147    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b591ff86-68e6-4d25-9fa3-2baafd863077" containerName="revoke-keys"
Nov 01 20:20:00.862292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862159    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f1b4fa09-c1da-4350-9efe-e06a2dc06f84" containerName="revoke-keys"
Nov 01 20:20:00.862292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862165    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b4fa09-c1da-4350-9efe-e06a2dc06f84" containerName="revoke-keys"
Nov 01 20:20:00.862292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862277    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b4fa09-c1da-4350-9efe-e06a2dc06f84" containerName="revoke-keys"
Nov 01 20:20:00.862406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.862295    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b591ff86-68e6-4d25-9fa3-2baafd863077" containerName="revoke-keys"
Nov 01 20:20:00.865459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.865438    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:00.869350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.869329    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:20:00.869585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.869564    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:20:00.869675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.869607    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:20:00.869770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.869754    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:20:00.888674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:00.888636    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"]
Nov 01 20:20:01.002395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.002343    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.002703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.002676    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4xz\" (UniqueName: \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.002863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.002742    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.104729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.104634    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.105248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.104905    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4xz\" (UniqueName: \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.105248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.104960    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.105354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.105342    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.107341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.107318    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.112195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.112171    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4xz\" (UniqueName: \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\") pod \"maas-api-revoke-sub-d6b11a20-qjtw6\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.226977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.226942    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:01.513761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:01.513732    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"]
Nov 01 20:20:01.514089 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:01.514069    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd55686_ab0f_4d04_bed7_f4adf1f12445.slice/crio-ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a WatchSource:0}: Error finding container ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a: Status 404 returned error can't find the container with id ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a
Nov 01 20:20:02.042952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.042894    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6" event={"ID":"7bd55686-ab0f-4d04-bed7-f4adf1f12445","Type":"ContainerStarted","Data":"6b3a9e32b0c5a3e560de8c3963561c028b38b2edac49f27426911308ca9ad344"}
Nov 01 20:20:02.043246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.043222    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6" event={"ID":"7bd55686-ab0f-4d04-bed7-f4adf1f12445","Type":"ContainerStarted","Data":"ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a"}
Nov 01 20:20:02.076312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.074863    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6" podStartSLOduration=2.074843048 podStartE2EDuration="2.074843048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:02.06757281 +0000 UTC m=+1289.788651486" watchObservedRunningTime="2026-11-01 20:20:02.074843048 +0000 UTC m=+1289.795921723"
Nov 01 20:20:02.970362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.970262    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"]
Nov 01 20:20:02.973764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.973740    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:02.983293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:02.983053    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"]
Nov 01 20:20:03.003807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.002119    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.003807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.002225    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.003807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.002284    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcdk\" (UniqueName: \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.055361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.055322    5041 generic.go:356] "Generic (PLEG): container finished" podID="7bd55686-ab0f-4d04-bed7-f4adf1f12445" containerID="6b3a9e32b0c5a3e560de8c3963561c028b38b2edac49f27426911308ca9ad344" exitCode=0
Nov 01 20:20:03.055564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.055387    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6" event={"ID":"7bd55686-ab0f-4d04-bed7-f4adf1f12445","Type":"ContainerDied","Data":"6b3a9e32b0c5a3e560de8c3963561c028b38b2edac49f27426911308ca9ad344"}
Nov 01 20:20:03.103873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.103818    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.104061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.103927    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.104061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.103978    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcdk\" (UniqueName: \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.104615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.104586    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.106746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.106722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.118255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.118219    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcdk\" (UniqueName: \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\") pod \"maas-api-revoke-sub-e29d210a-zj5cj\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.294392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.294351    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:03.583821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:03.583052    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"]
Nov 01 20:20:04.060125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.060093    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:20:04.065935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.065891    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj" event={"ID":"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae","Type":"ContainerStarted","Data":"c03ce3cabd592b1d68ceba28e7d2b28c4e8e7ee84ff0aa95af6460b591152301"}
Nov 01 20:20:04.066093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.065983    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj" event={"ID":"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae","Type":"ContainerStarted","Data":"0282832c011acb4fb7d7d8d55f5d0fc4e04709c81920e176f964aa0abc3f4cbc"}
Nov 01 20:20:04.067855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.067829    5041 generic.go:356] "Generic (PLEG): container finished" podID="108e9100-17c0-473e-9605-800b410f5d61" containerID="985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5" exitCode=0
Nov 01 20:20:04.067999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.067877    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"
Nov 01 20:20:04.067999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.067904    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerDied","Data":"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"}
Nov 01 20:20:04.067999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.067950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw" event={"ID":"108e9100-17c0-473e-9605-800b410f5d61","Type":"ContainerDied","Data":"a954c400842806cebfded060467b2e8bc845c59b73d63545522d0c587b7c3fc9"}
Nov 01 20:20:04.067999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.067972    5041 scope.go:122] "RemoveContainer" containerID="985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"
Nov 01 20:20:04.079886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.079857    5041 scope.go:122] "RemoveContainer" containerID="e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"
Nov 01 20:20:04.091063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.091028    5041 scope.go:122] "RemoveContainer" containerID="985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"
Nov 01 20:20:04.091321 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:04.091302    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5\": container with ID starting with 985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5 not found: ID does not exist" containerID="985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"
Nov 01 20:20:04.091390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.091331    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5"} err="failed to get container status \"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5\": rpc error: code = NotFound desc = could not find container \"985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5\": container with ID starting with 985f3c6dec7394f11276de5ddec330ed9a109909f56aad0c35b86135246012e5 not found: ID does not exist"
Nov 01 20:20:04.091390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.091349    5041 scope.go:122] "RemoveContainer" containerID="e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"
Nov 01 20:20:04.091585 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:04.091569    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53\": container with ID starting with e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53 not found: ID does not exist" containerID="e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"
Nov 01 20:20:04.091667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.091589    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53"} err="failed to get container status \"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53\": rpc error: code = NotFound desc = could not find container \"e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53\": container with ID starting with e537fda35c346cfdb2fd1cde7fd547f694436be67dde1e7316c916c3b450ef53 not found: ID does not exist"
Nov 01 20:20:04.121821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.121699    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj" podStartSLOduration=2.121682538 podStartE2EDuration="2.121682538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:04.117254967 +0000 UTC m=+1291.838333659" watchObservedRunningTime="2026-11-01 20:20:04.121682538 +0000 UTC m=+1291.842761215"
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141439    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141563    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141605    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141701    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141704    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141806    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141911    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.141983    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142017    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142064    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\") pod \"108e9100-17c0-473e-9605-800b410f5d61\" (UID: \"108e9100-17c0-473e-9605-800b410f5d61\") "
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142089    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142281    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142497    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142928    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142947    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142962    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108e9100-17c0-473e-9605-800b410f5d61-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.142977    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.143872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.143454    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:04.146512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.146484    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:04.147694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.147673    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:04.148749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.148717    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "kube-api-access-v96fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:04.148749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.148731    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo" pod "108e9100-17c0-473e-9605-800b410f5d61" (UID: "108e9100-17c0-473e-9605-800b410f5d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:20:04.243926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.243889    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.243926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.243927    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96fv\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-kube-api-access-v96fv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.244107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.243943    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108e9100-17c0-473e-9605-800b410f5d61-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.244107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.243955    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108e9100-17c0-473e-9605-800b410f5d61-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.244107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.243967    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108e9100-17c0-473e-9605-800b410f5d61-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.273663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.273638    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:04.344897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.344864    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\") pod \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") "
Nov 01 20:20:04.345092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.344992    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\") pod \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") "
Nov 01 20:20:04.345092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.345025    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\" (UniqueName: \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\") pod \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\" (UID: \"7bd55686-ab0f-4d04-bed7-f4adf1f12445\") "
Nov 01 20:20:04.345290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.345259    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca" pod "7bd55686-ab0f-4d04-bed7-f4adf1f12445" (UID: "7bd55686-ab0f-4d04-bed7-f4adf1f12445"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:04.345875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.345849    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.347256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.347233    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca" pod "7bd55686-ab0f-4d04-bed7-f4adf1f12445" (UID: "7bd55686-ab0f-4d04-bed7-f4adf1f12445"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:04.347367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.347348    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz" pod "7bd55686-ab0f-4d04-bed7-f4adf1f12445" (UID: "7bd55686-ab0f-4d04-bed7-f4adf1f12445"). InnerVolumeSpecName "kube-api-access-2f4xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:04.397475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.397430    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:20:04.402203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.402177    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3de00d-openshift-default-fb4cd864f-z2nrw"]
Nov 01 20:20:04.431480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.431440    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108e9100-17c0-473e-9605-800b410f5d61" path="/var/lib/kubelet/pods/108e9100-17c0-473e-9605-800b410f5d61/volumes"
Nov 01 20:20:04.447636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.447613    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd55686-ab0f-4d04-bed7-f4adf1f12445-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:04.447636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:04.447636    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4xz\" (UniqueName: \"kubernetes.io/projected/7bd55686-ab0f-4d04-bed7-f4adf1f12445-kube-api-access-2f4xz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:05.151828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:05.148073    5041 generic.go:356] "Generic (PLEG): container finished" podID="9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" containerID="c03ce3cabd592b1d68ceba28e7d2b28c4e8e7ee84ff0aa95af6460b591152301" exitCode=0
Nov 01 20:20:05.151828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:05.148211    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj" event={"ID":"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae","Type":"ContainerDied","Data":"c03ce3cabd592b1d68ceba28e7d2b28c4e8e7ee84ff0aa95af6460b591152301"}
Nov 01 20:20:05.178801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:05.157212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6" event={"ID":"7bd55686-ab0f-4d04-bed7-f4adf1f12445","Type":"ContainerDied","Data":"ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a"}
Nov 01 20:20:05.178801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:05.157266    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed48dee51d384028dc0659e597235fc77fabc585fc3ce28e611346796fdb392a"
Nov 01 20:20:05.178801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:05.157333    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"
Nov 01 20:20:06.172354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.172316    5041 generic.go:356] "Generic (PLEG): container finished" podID="d60f81ca-b7da-4b99-899c-f8ba5651ffa2" containerID="7201b96ecd21d468b4568bd957fb65105834d2a831f3736fa70ddb52d8a053a6" exitCode=0
Nov 01 20:20:06.172891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.172392    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr" event={"ID":"d60f81ca-b7da-4b99-899c-f8ba5651ffa2","Type":"ContainerDied","Data":"7201b96ecd21d468b4568bd957fb65105834d2a831f3736fa70ddb52d8a053a6"}
Nov 01 20:20:06.362382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.362351    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:06.514891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.514806    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\") pod \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") "
Nov 01 20:20:06.515047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.515034    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\") pod \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") "
Nov 01 20:20:06.515096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.515066    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\" (UniqueName: \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\") pod \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\" (UID: \"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae\") "
Nov 01 20:20:06.515154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.515106    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca" pod "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" (UID: "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:06.515988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.515961    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:06.517349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.517322    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk" pod "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" (UID: "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae"). InnerVolumeSpecName "kube-api-access-ztcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:06.517474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.517433    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca" pod "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" (UID: "9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:06.620802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.617653    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:06.620802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:06.617704    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcdk\" (UniqueName: \"kubernetes.io/projected/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae-kube-api-access-ztcdk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:07.183053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.183019    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"
Nov 01 20:20:07.183053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.183017    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj" event={"ID":"9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae","Type":"ContainerDied","Data":"0282832c011acb4fb7d7d8d55f5d0fc4e04709c81920e176f964aa0abc3f4cbc"}
Nov 01 20:20:07.183581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.183111    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0282832c011acb4fb7d7d8d55f5d0fc4e04709c81920e176f964aa0abc3f4cbc"
Nov 01 20:20:07.373644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.372610    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:20:07.538389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538362    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\" (UniqueName: \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\") pod \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") "
Nov 01 20:20:07.538576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538470    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\") pod \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") "
Nov 01 20:20:07.538576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538515    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\") pod \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") "
Nov 01 20:20:07.538709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538651    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\") pod \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\" (UID: \"d60f81ca-b7da-4b99-899c-f8ba5651ffa2\") "
Nov 01 20:20:07.538901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538875    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path" pod "d60f81ca-b7da-4b99-899c-f8ba5651ffa2" (UID: "d60f81ca-b7da-4b99-899c-f8ba5651ffa2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:07.539000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.538962    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle" pod "d60f81ca-b7da-4b99-899c-f8ba5651ffa2" (UID: "d60f81ca-b7da-4b99-899c-f8ba5651ffa2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:07.539071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.539053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle" pod "d60f81ca-b7da-4b99-899c-f8ba5651ffa2" (UID: "d60f81ca-b7da-4b99-899c-f8ba5651ffa2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:07.539375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.539344    5041 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-archives-path\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:07.539375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.539376    5041 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-service-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:07.539804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.539392    5041 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-trusted-ca-bundle\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:07.540495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.540474    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp" pod "d60f81ca-b7da-4b99-899c-f8ba5651ffa2" (UID: "d60f81ca-b7da-4b99-899c-f8ba5651ffa2"). InnerVolumeSpecName "kube-api-access-srspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:07.640624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:07.640590    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srspp\" (UniqueName: \"kubernetes.io/projected/d60f81ca-b7da-4b99-899c-f8ba5651ffa2-kube-api-access-srspp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:08.193322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:08.193295    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr"
Nov 01 20:20:08.193743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:08.193337    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-c5hj4-6c5dr" event={"ID":"d60f81ca-b7da-4b99-899c-f8ba5651ffa2","Type":"ContainerDied","Data":"4edcbc87fb677bcc5dab7a78f7515c15ddd11fe4b94059db0c7ee9e9487ee4b7"}
Nov 01 20:20:08.193743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:08.193382    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4edcbc87fb677bcc5dab7a78f7515c15ddd11fe4b94059db0c7ee9e9487ee4b7"
Nov 01 20:20:15.751889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.751757    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:20:15.754086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754056    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d60f81ca-b7da-4b99-899c-f8ba5651ffa2" containerName="insights-gathering"
Nov 01 20:20:15.754086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754086    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60f81ca-b7da-4b99-899c-f8ba5651ffa2" containerName="insights-gathering"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754143    5041 cpu_manager.go:414] "containerMap: removing container" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754151    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754166    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7bd55686-ab0f-4d04-bed7-f4adf1f12445" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754172    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd55686-ab0f-4d04-bed7-f4adf1f12445" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754191    5041 cpu_manager.go:414] "containerMap: removing container" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754197    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754206    5041 cpu_manager.go:414] "containerMap: removing container" podUID="9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754212    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754369    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754382    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd55686-ab0f-4d04-bed7-f4adf1f12445" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754396    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="108e9100-17c0-473e-9605-800b410f5d61" containerName="istio-proxy"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754404    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" containerName="revoke-keys"
Nov 01 20:20:15.755246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.754420    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60f81ca-b7da-4b99-899c-f8ba5651ffa2" containerName="insights-gathering"
Nov 01 20:20:15.760447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.760418    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.765254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.764308    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:20:15.780854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.775004    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-933245-openshift-default-dockercfg-2zh47\""
Nov 01 20:20:15.933606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.933566    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.934373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.934343    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.934557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.934540    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.934709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.934692    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqg2s\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.934848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.934833    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.935036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.935018    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.935233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.935217    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.935404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.935387    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:15.935514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:15.935500    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.037650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037614    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqg2s\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.037868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037658    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.037868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037714    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.037868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037787    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.037868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037862    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.038099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.037884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.038099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.038013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.038099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.038052    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.038099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.038077    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.038928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.038900    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.039259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.039208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.039507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.039481    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.039838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.039705    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.040288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.040256    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.042298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.042270    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.042657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.042632    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.048676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.048222    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqg2s\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.060815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.052422    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\") pod \"e2e-autores-933245-openshift-default-8b8b4856c-nbw24\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") " pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.088292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.088255    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:16.389387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.389351    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:20:16.389960 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:16.389933    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a067d7_7f7b_4f13_b9af_f6358c85ba1b.slice/crio-6d7bce1a43c68f003c7e9f073773e281568fbab067df0ff62f36ab8ab41da787 WatchSource:0}: Error finding container 6d7bce1a43c68f003c7e9f073773e281568fbab067df0ff62f36ab8ab41da787: Status 404 returned error can't find the container with id 6d7bce1a43c68f003c7e9f073773e281568fbab067df0ff62f36ab8ab41da787
Nov 01 20:20:16.393205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.393157    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:16.393288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.393221    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:16.393288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:16.393253    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:17.348241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:17.348202    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerStarted","Data":"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"}
Nov 01 20:20:17.348635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:17.348260    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerStarted","Data":"6d7bce1a43c68f003c7e9f073773e281568fbab067df0ff62f36ab8ab41da787"}
Nov 01 20:20:17.370186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:17.370129    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podStartSLOduration=2.370114897 podStartE2EDuration="2.370114897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:17.369587601 +0000 UTC m=+1305.090666277" watchObservedRunningTime="2026-11-01 20:20:17.370114897 +0000 UTC m=+1305.091193573"
Nov 01 20:20:18.088754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.088715    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:18.095557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.095531    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:18.360772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.360635    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:18.362761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.362723    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:20:18.480711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.480648    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:20:18.491645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.491616    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:20:18.492193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:18.492161    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy" containerID="cri-o://690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4" gracePeriod=30
Nov 01 20:20:19.055751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.055714    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:20:19.059988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.059965    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.066830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.066760    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-aaea41-openshift-default-dockercfg-7lvtz\""
Nov 01 20:20:19.073788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.073747    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:20:19.193868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.193835    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.193914    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.193980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194020    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194044    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194120    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznx7\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194190    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194240    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.194346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.194334    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296318    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296387    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296472    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296514    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296544    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296588    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznx7\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296653    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.296805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.297100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296898    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.297100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.296935    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.297100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.297019    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.297262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.297180    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.297505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.297481    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.299274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.299251    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.300206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.300186    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.305197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.305176    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.305332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.305314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznx7\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\") pod \"e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") " pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.362916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.362822    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:20:19.378212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.378177    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:19.714177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.714147    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:20:19.718378 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:19.718343    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5407d2_caa7_4795_a92a_df01bd5a8e9b.slice/crio-44281c1d442d0b2a77c0a8dedc9f99103fdf67ffeb16282faf55728a52e00378 WatchSource:0}: Error finding container 44281c1d442d0b2a77c0a8dedc9f99103fdf67ffeb16282faf55728a52e00378: Status 404 returned error can't find the container with id 44281c1d442d0b2a77c0a8dedc9f99103fdf67ffeb16282faf55728a52e00378
Nov 01 20:20:19.724073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.721956    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:19.724073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.722042    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:19.724073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:19.722089    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:20.416862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:20.416820    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" event={"ID":"5f5407d2-caa7-4795-a92a-df01bd5a8e9b","Type":"ContainerStarted","Data":"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"}
Nov 01 20:20:20.417361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:20.416915    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" event={"ID":"5f5407d2-caa7-4795-a92a-df01bd5a8e9b","Type":"ContainerStarted","Data":"44281c1d442d0b2a77c0a8dedc9f99103fdf67ffeb16282faf55728a52e00378"}
Nov 01 20:20:20.438409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:20.438350    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podStartSLOduration=1.438336423 podStartE2EDuration="1.438336423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:20.436446021 +0000 UTC m=+1308.157524701" watchObservedRunningTime="2026-11-01 20:20:20.438336423 +0000 UTC m=+1308.159415102"
Nov 01 20:20:21.379142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:21.379106    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:21.381278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:21.381244    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:21.381459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:21.381298    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:22.379999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:22.378836    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:22.379999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:22.378896    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:23.384201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:23.384140    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:23.384765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:23.384735    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:24.293035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.293007    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:20:24.370547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.370510    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" podUID="1ef565c9-e4af-46b3-ab54-66af348e0872" containerName="payload-processing" containerID="cri-o://e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b" gracePeriod=25
Nov 01 20:20:24.379459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.379408    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:24.379630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.379475    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585315    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585423    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585563    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585637    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585706    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585745    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585868    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.592302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.585965    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\") pod \"525af09b-2ace-4907-b979-2e9439ce47a0\" (UID: \"525af09b-2ace-4907-b979-2e9439ce47a0\") "
Nov 01 20:20:24.663939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.598913    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:24.663939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.617711    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:24.663939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.618946    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:24.663939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.619160    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:24.663939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.621556    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:24.677483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.675887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702459    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702495    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702535    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702551    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702565    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/525af09b-2ace-4907-b979-2e9439ce47a0-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.705600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.702578    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/525af09b-2ace-4907-b979-2e9439ce47a0-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.721551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.721515    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:24.724094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.724064    5041 generic.go:356] "Generic (PLEG): container finished" podID="525af09b-2ace-4907-b979-2e9439ce47a0" containerID="690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4" exitCode=0
Nov 01 20:20:24.724685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.724657    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerDied","Data":"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"}
Nov 01 20:20:24.724938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.724917    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb" event={"ID":"525af09b-2ace-4907-b979-2e9439ce47a0","Type":"ContainerDied","Data":"83dff9ed927dfb5525059326c02bad8dd8cf2d60aecf443057de768cd9ad86d7"}
Nov 01 20:20:24.728623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.725066    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"
Nov 01 20:20:24.728885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.725083    5041 scope.go:122] "RemoveContainer" containerID="690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"
Nov 01 20:20:24.728987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.726425    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:20:24.729047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.726502    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j" pod "525af09b-2ace-4907-b979-2e9439ce47a0" (UID: "525af09b-2ace-4907-b979-2e9439ce47a0"). InnerVolumeSpecName "kube-api-access-bq54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:24.787214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.787180    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:20:24.790755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.788672    5041 scope.go:122] "RemoveContainer" containerID="fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"
Nov 01 20:20:24.792828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.792711    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6afe1a-openshift-default-ff85dbf79-zqrcb"]
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.802592    5041 scope.go:122] "RemoveContainer" containerID="690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:24.802942    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4\": container with ID starting with 690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4 not found: ID does not exist" containerID="690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.802970    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4"} err="failed to get container status \"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4\": rpc error: code = NotFound desc = could not find container \"690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4\": container with ID starting with 690c257034a3ef7963c3d5ef4c265f17ea3a731f4c3a8eaf3d8e3e5c5e8754e4 not found: ID does not exist"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.802991    5041 scope.go:122] "RemoveContainer" containerID="fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:24.803195    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40\": container with ID starting with fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40 not found: ID does not exist" containerID="fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.803212    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40"} err="failed to get container status \"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40\": rpc error: code = NotFound desc = could not find container \"fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40\": container with ID starting with fcc3b06bfbf3b5b0d53116fce7757c866274a7619856357e35cfaf330ade9e40 not found: ID does not exist"
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.803671    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.803696    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/525af09b-2ace-4907-b979-2e9439ce47a0-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:24.803732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:24.803710    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq54j\" (UniqueName: \"kubernetes.io/projected/525af09b-2ace-4907-b979-2e9439ce47a0-kube-api-access-bq54j\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:25.025251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.025225    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:20:25.140331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.140300    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\") pod \"1ef565c9-e4af-46b3-ab54-66af348e0872\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") "
Nov 01 20:20:25.140496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.140336    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\") pod \"1ef565c9-e4af-46b3-ab54-66af348e0872\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") "
Nov 01 20:20:25.140496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.140419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\") pod \"1ef565c9-e4af-46b3-ab54-66af348e0872\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") "
Nov 01 20:20:25.140496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.140448    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\") pod \"1ef565c9-e4af-46b3-ab54-66af348e0872\" (UID: \"1ef565c9-e4af-46b3-ab54-66af348e0872\") "
Nov 01 20:20:25.140765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.140734    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume" pod "1ef565c9-e4af-46b3-ab54-66af348e0872" (UID: "1ef565c9-e4af-46b3-ab54-66af348e0872"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:25.141267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.141010    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume" pod "1ef565c9-e4af-46b3-ab54-66af348e0872" (UID: "1ef565c9-e4af-46b3-ab54-66af348e0872"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:25.141267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.141244    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:25.141267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.141267    5041 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ef565c9-e4af-46b3-ab54-66af348e0872-routing-overlay-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:25.175439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.175411    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr" pod "1ef565c9-e4af-46b3-ab54-66af348e0872" (UID: "1ef565c9-e4af-46b3-ab54-66af348e0872"). InnerVolumeSpecName "kube-api-access-tbjjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:25.176172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.176145    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials" pod "1ef565c9-e4af-46b3-ab54-66af348e0872" (UID: "1ef565c9-e4af-46b3-ab54-66af348e0872"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:25.268855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.264064    5041 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-provider-credentials\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:25.268855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.264105    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbjjr\" (UniqueName: \"kubernetes.io/projected/1ef565c9-e4af-46b3-ab54-66af348e0872-kube-api-access-tbjjr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:25.380802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.380003    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:25.380802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.380081    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:25.736062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.736014    5041 generic.go:356] "Generic (PLEG): container finished" podID="1ef565c9-e4af-46b3-ab54-66af348e0872" containerID="e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b" exitCode=0
Nov 01 20:20:25.736563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.736067    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"
Nov 01 20:20:25.736563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.736107    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" event={"ID":"1ef565c9-e4af-46b3-ab54-66af348e0872","Type":"ContainerDied","Data":"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"}
Nov 01 20:20:25.736563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.736171    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt" event={"ID":"1ef565c9-e4af-46b3-ab54-66af348e0872","Type":"ContainerDied","Data":"6bfedfaa3419a67f222e97329cec4e5bddfa612301ea6208a79a621c5aa4ba0f"}
Nov 01 20:20:25.736563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.736194    5041 scope.go:122] "RemoveContainer" containerID="e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"
Nov 01 20:20:25.774866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.774829    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:20:25.775986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.775963    5041 scope.go:122] "RemoveContainer" containerID="e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"
Nov 01 20:20:25.776375 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:25.776349    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b\": container with ID starting with e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b not found: ID does not exist" containerID="e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"
Nov 01 20:20:25.776472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.776381    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b"} err="failed to get container status \"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b\": rpc error: code = NotFound desc = could not find container \"e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b\": container with ID starting with e48300f166b03db74263bc33d098bee8df88cb966dc530772477ccf2fcfacd1b not found: ID does not exist"
Nov 01 20:20:25.779228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:25.779205    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5fg8dt"]
Nov 01 20:20:26.379343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:26.379307    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:26.379535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:26.379366    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:26.430740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:26.430709    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef565c9-e4af-46b3-ab54-66af348e0872" path="/var/lib/kubelet/pods/1ef565c9-e4af-46b3-ab54-66af348e0872/volumes"
Nov 01 20:20:26.431297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:26.431280    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" path="/var/lib/kubelet/pods/525af09b-2ace-4907-b979-2e9439ce47a0/volumes"
Nov 01 20:20:27.383214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:27.383177    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 01 20:20:27.383826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:27.383792    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 01 20:20:28.655803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:28.655300    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.018858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.017930    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.051212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.051143    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:20:29.247106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.247056    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.247901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.247859    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.267005 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:29.250254    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:20:29.356804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.355755    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.385765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.385724    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:20:29.386066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.386033    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.421183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.421148    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:29.450909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.450871    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:29.451543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.451430    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:29.470071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:29.470043    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:20:30.055553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.055511    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:20:30.056669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.056644    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:20:30.094802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.061084    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": context deadline exceeded"
Nov 01 20:20:30.094802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.076465    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:20:30.225893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.225861    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:20:30.226168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:30.226149    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:20:33.517870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.517819    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main" containerID="cri-o://33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e" gracePeriod=45
Nov 01 20:20:33.813672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.813268    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:20:33.918834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.918796    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.918858    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.918889    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.918973    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\" (UniqueName: \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919052    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:33.919207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919087    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919197    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:33.919351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919312    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\") pod \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\" (UID: \"ee8fd09e-a178-4f2c-abfb-cbe568d2e489\") "
Nov 01 20:20:33.919351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919341    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:33.919956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919923    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:33.919956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919949    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:33.920144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.919965    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:33.921326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.921304    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:20:33.921472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.921455    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "kube-api-access-qzztl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:33.921902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:33.921879    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs" pod "ee8fd09e-a178-4f2c-abfb-cbe568d2e489" (UID: "ee8fd09e-a178-4f2c-abfb-cbe568d2e489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:34.036864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.036059    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:34.036864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.036095    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:34.036864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.036109    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzztl\" (UniqueName: \"kubernetes.io/projected/ee8fd09e-a178-4f2c-abfb-cbe568d2e489-kube-api-access-qzztl\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:34.229816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.229748    5041 generic.go:356] "Generic (PLEG): container finished" podID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerID="33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e" exitCode=0
Nov 01 20:20:34.229816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.229811    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"
Nov 01 20:20:34.230024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.229831    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" event={"ID":"ee8fd09e-a178-4f2c-abfb-cbe568d2e489","Type":"ContainerDied","Data":"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"}
Nov 01 20:20:34.230024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.229887    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp" event={"ID":"ee8fd09e-a178-4f2c-abfb-cbe568d2e489","Type":"ContainerDied","Data":"213c5f1b8ff93cfe95f7b9f1d6389d72e4ace6533f1f6f188215e6c9dc8b66e0"}
Nov 01 20:20:34.230024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.229910    5041 scope.go:122] "RemoveContainer" containerID="33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"
Nov 01 20:20:34.241752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.241735    5041 scope.go:122] "RemoveContainer" containerID="33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"
Nov 01 20:20:34.242076 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:34.242058    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e\": container with ID starting with 33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e not found: ID does not exist" containerID="33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"
Nov 01 20:20:34.242115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.242086    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e"} err="failed to get container status \"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e\": rpc error: code = NotFound desc = could not find container \"33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e\": container with ID starting with 33728b6ef0982cf286756cf2f2392e328e803cd857c0b182c576a77684f4533e not found: ID does not exist"
Nov 01 20:20:34.275014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.274943    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:20:34.282238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.282205    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-e83b4b/auth-test-model-e83b4b-kserve-5b6cd8df97-hdfkp"]
Nov 01 20:20:34.430618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:34.430536    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" path="/var/lib/kubelet/pods/ee8fd09e-a178-4f2c-abfb-cbe568d2e489/volumes"
Nov 01 20:20:35.031580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:35.031543    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"]
Nov 01 20:20:35.035750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:35.035721    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66f7552d-2djwt"]
Nov 01 20:20:35.145424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:35.145394    5041 scope.go:122] "RemoveContainer" containerID="de4f593ac7be33ca239cd3b55ee311d4d776e40a327d03a7a7533b41d266c879"
Nov 01 20:20:35.157052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:35.157024    5041 scope.go:122] "RemoveContainer" containerID="b8b4fcad66772ed02bd69cef3614940e352a893a88e677b32070432cc982ef24"
Nov 01 20:20:35.169660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:35.169587    5041 scope.go:122] "RemoveContainer" containerID="be16ce89683719de23d3b2d8efa148406e99c7edd8c392afe40ed1f486516b99"
Nov 01 20:20:36.430870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:36.430834    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271d2f68-4d40-42b4-8b6a-6dabc3fe3401" path="/var/lib/kubelet/pods/271d2f68-4d40-42b4-8b6a-6dabc3fe3401/volumes"
Nov 01 20:20:37.488737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.488703    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"]
Nov 01 20:20:37.489382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489367    5041 cpu_manager.go:414] "containerMap: removing container" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:37.489382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489382    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:37.489536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489402    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1ef565c9-e4af-46b3-ab54-66af348e0872" containerName="payload-processing"
Nov 01 20:20:37.489536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489407    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef565c9-e4af-46b3-ab54-66af348e0872" containerName="payload-processing"
Nov 01 20:20:37.489536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489429    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main"
Nov 01 20:20:37.489536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489435    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main"
Nov 01 20:20:37.489536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489532    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8fd09e-a178-4f2c-abfb-cbe568d2e489" containerName="main"
Nov 01 20:20:37.489860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489548    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:37.489860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489556    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef565c9-e4af-46b3-ab54-66af348e0872" containerName="payload-processing"
Nov 01 20:20:37.489860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.489563    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:37.495610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.495588    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.499092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.499066    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:20:37.500250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.500226    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:20:37.500377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.500299    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:20:37.500439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.500394    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:20:37.501064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.501042    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"]
Nov 01 20:20:37.601461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.601421    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742jf\" (UniqueName: \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.601632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.601583    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.601723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.601704    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.705079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.703543    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.705079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.703899    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742jf\" (UniqueName: \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.705079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.704048    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.705578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.705547    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.747900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.726445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742jf\" (UniqueName: \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.747900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.743038    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fba7223-bjfsl\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:37.823802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:37.818274    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:38.176724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:38.176698    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"]
Nov 01 20:20:38.176996 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:38.176971    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3a1957_093b_46e1_af08_ece65b0e04ad.slice/crio-c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7 WatchSource:0}: Error finding container c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7: Status 404 returned error can't find the container with id c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7
Nov 01 20:20:38.296840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:38.296803    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl" event={"ID":"7d3a1957-093b-46e1-af08-ece65b0e04ad","Type":"ContainerStarted","Data":"c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7"}
Nov 01 20:20:39.307310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:39.307215    5041 generic.go:356] "Generic (PLEG): container finished" podID="7d3a1957-093b-46e1-af08-ece65b0e04ad" containerID="c5993555db9898b077f8ce7f4fa694537112a0784a1b4fa4b84a3330cf78cdd1" exitCode=0
Nov 01 20:20:39.307704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:39.307303    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl" event={"ID":"7d3a1957-093b-46e1-af08-ece65b0e04ad","Type":"ContainerDied","Data":"c5993555db9898b077f8ce7f4fa694537112a0784a1b4fa4b84a3330cf78cdd1"}
Nov 01 20:20:40.498283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.498251    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:40.559516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.559479    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\") pod \"7d3a1957-093b-46e1-af08-ece65b0e04ad\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") "
Nov 01 20:20:40.559681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.559557    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\") pod \"7d3a1957-093b-46e1-af08-ece65b0e04ad\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") "
Nov 01 20:20:40.559876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.559852    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca" pod "7d3a1957-093b-46e1-af08-ece65b0e04ad" (UID: "7d3a1957-093b-46e1-af08-ece65b0e04ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:40.559963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.559925    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\" (UniqueName: \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\") pod \"7d3a1957-093b-46e1-af08-ece65b0e04ad\" (UID: \"7d3a1957-093b-46e1-af08-ece65b0e04ad\") "
Nov 01 20:20:40.560680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.560657    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:40.562104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.562077    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca" pod "7d3a1957-093b-46e1-af08-ece65b0e04ad" (UID: "7d3a1957-093b-46e1-af08-ece65b0e04ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:40.562822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.562793    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf" pod "7d3a1957-093b-46e1-af08-ece65b0e04ad" (UID: "7d3a1957-093b-46e1-af08-ece65b0e04ad"). InnerVolumeSpecName "kube-api-access-742jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:40.662679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.662638    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742jf\" (UniqueName: \"kubernetes.io/projected/7d3a1957-093b-46e1-af08-ece65b0e04ad-kube-api-access-742jf\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:40.662679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:40.662681    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d3a1957-093b-46e1-af08-ece65b0e04ad-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:41.023312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:41.023227    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"]
Nov 01 20:20:41.027277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:41.027251    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84612ea4-xlhcc"]
Nov 01 20:20:41.324167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:41.324135    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"
Nov 01 20:20:41.324341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:41.324134    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl" event={"ID":"7d3a1957-093b-46e1-af08-ece65b0e04ad","Type":"ContainerDied","Data":"c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7"}
Nov 01 20:20:41.324341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:41.324258    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c41d3519e302f520c11917c62d2c1d3430a1b4e136f71679d3ebb4f00e57dda7"
Nov 01 20:20:42.429385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:42.429344    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8" path="/var/lib/kubelet/pods/fa1b925d-4f17-4a37-bf8a-ea3cc8e5a6f8/volumes"
Nov 01 20:20:47.022089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:47.022054    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"]
Nov 01 20:20:47.026255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:47.026228    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a162ac60-bk5nz"]
Nov 01 20:20:48.464802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.464597    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf721fd-7f50-4aab-97a3-c9f756c9728d" path="/var/lib/kubelet/pods/1cf721fd-7f50-4aab-97a3-c9f756c9728d/volumes"
Nov 01 20:20:48.852799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.843222    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:20:48.865835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.854077    5041 cpu_manager.go:414] "containerMap: removing container" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:48.866094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.866069    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525af09b-2ace-4907-b979-2e9439ce47a0" containerName="istio-proxy"
Nov 01 20:20:48.866253 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.866243    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7d3a1957-093b-46e1-af08-ece65b0e04ad" containerName="revoke-keys"
Nov 01 20:20:48.866324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.866315    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3a1957-093b-46e1-af08-ece65b0e04ad" containerName="revoke-keys"
Nov 01 20:20:48.866619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.866605    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3a1957-093b-46e1-af08-ece65b0e04ad" containerName="revoke-keys"
Nov 01 20:20:48.873142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.873094    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:20:48.873472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.873448    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.884940    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.885663    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.885748    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.885803    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-3f909928dd00\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.885861    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-8876b9054304\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.885938    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.886024    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.886057    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-85078fdb41c3\""
Nov 01 20:20:48.893803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.886089    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\""
Nov 01 20:20:48.951052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.951009    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:48.951373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.951349    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:48.951661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.951644    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:48.951831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:48.951814    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27csj\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.070832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.070757    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.071156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.071132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27csj\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.071376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.071355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.071553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.071536    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.072752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.072727    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.073491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.073469    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.082690 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.082659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.087455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.087427    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27csj\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.254090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.253602    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:49.458611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.458582    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"]
Nov 01 20:20:49.495830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.480919    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"]
Nov 01 20:20:49.497951 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.494035    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.504728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.504666    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:20:49.504908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.504840    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:20:49.571699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.571493    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:20:49.571699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.571637    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:20:49.595703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.595274    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.595703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.595548    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d555l\" (UniqueName: \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.595703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.595584    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.697131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.696963    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d555l\" (UniqueName: \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.697131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.697021    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.697131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.697082    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.698410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.698256    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.708040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.706607    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.710851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.710800    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d555l\" (UniqueName: \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\") pod \"maas-api-revoke-sub-41d5051e-gtbmt\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:49.739950 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:49.739919    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea0a61b_bbdf_43fa_9c98_2a6ea272dfc2.slice/crio-c6f9afe70fb123f9ef35d0481840e21149b50e5237ecc7df4872e9d15ea8132d WatchSource:0}: Error finding container c6f9afe70fb123f9ef35d0481840e21149b50e5237ecc7df4872e9d15ea8132d: Status 404 returned error can't find the container with id c6f9afe70fb123f9ef35d0481840e21149b50e5237ecc7df4872e9d15ea8132d
Nov 01 20:20:49.741098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.741077    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:20:49.827349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:49.827212    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:50.275295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.274543    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"]
Nov 01 20:20:50.485675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.485640    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" event={"ID":"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2","Type":"ContainerStarted","Data":"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"}
Nov 01 20:20:50.485897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.485705    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" event={"ID":"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2","Type":"ContainerStarted","Data":"c6f9afe70fb123f9ef35d0481840e21149b50e5237ecc7df4872e9d15ea8132d"}
Nov 01 20:20:50.485897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.485740    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:20:50.487439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.487406    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt" event={"ID":"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3","Type":"ContainerStarted","Data":"9e7711df81ea0499d17f5d283472b10ea60197cc341672d382a834d97d77337e"}
Nov 01 20:20:50.506864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:50.506754    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" podStartSLOduration=2.506739176 podStartE2EDuration="2.506739176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:50.502282744 +0000 UTC m=+1338.223361421" watchObservedRunningTime="2026-11-01 20:20:50.506739176 +0000 UTC m=+1338.227817853"
Nov 01 20:20:51.296765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.296734    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:20:51.305758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.305727    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:20:51.305930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.305904    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.310772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.310742    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-933245\"/\"default-dockercfg-wdlqn\""
Nov 01 20:20:51.310946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.310857    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-933245\"/\"openshift-service-ca.crt\""
Nov 01 20:20:51.311133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.310982    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-933245\"/\"autores-model-933245-kserve-self-signed-certs\""
Nov 01 20:20:51.311133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.311063    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-933245\"/\"kube-root-ca.crt\""
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.365331    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8nmf\" (UniqueName: \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.365386    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.365415    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.367333    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.367370    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.367804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.367411    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501146    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501212    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501398    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8nmf\" (UniqueName: \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501420    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501434    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.504804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.501981    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.533265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.524187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.609218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.609182    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.620390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.618192    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt" event={"ID":"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3","Type":"ContainerStarted","Data":"8ac9d6f3bdd745d944986c1a8233a4a051afd7d55166d17a01c164edd2526682"}
Nov 01 20:20:51.629847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.629747    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.634381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.634212    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8nmf\" (UniqueName: \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.634894 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.634754    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\") pod \"autores-model-933245-kserve-7c7f8468f7-62526\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") " pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:51.669801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.647620    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt" podStartSLOduration=2.647602667 podStartE2EDuration="2.647602667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:51.640546999 +0000 UTC m=+1339.361625687" watchObservedRunningTime="2026-11-01 20:20:51.647602667 +0000 UTC m=+1339.368681344"
Nov 01 20:20:51.924355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:51.924273    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:52.235414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.233798    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:20:52.629274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.629239    5041 generic.go:356] "Generic (PLEG): container finished" podID="b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" containerID="8ac9d6f3bdd745d944986c1a8233a4a051afd7d55166d17a01c164edd2526682" exitCode=0
Nov 01 20:20:52.629748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.629332    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt" event={"ID":"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3","Type":"ContainerDied","Data":"8ac9d6f3bdd745d944986c1a8233a4a051afd7d55166d17a01c164edd2526682"}
Nov 01 20:20:52.631210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.631186    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" event={"ID":"33924fab-f553-4b02-b482-dd95d1c49f30","Type":"ContainerStarted","Data":"e5340d73c5e03362e422e48a3463286608f136a38a3d868c4ec7b9fb5dd1390e"}
Nov 01 20:20:52.631324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.631220    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" event={"ID":"33924fab-f553-4b02-b482-dd95d1c49f30","Type":"ContainerStarted","Data":"d4e90ce472ff4ec25b6caac8e0823f373db008c3d7a42a9adbd0b18bbd61dcca"}
Nov 01 20:20:52.691748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.691672    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" podStartSLOduration=1.6916515269999999 podStartE2EDuration="1.691651527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:52.688379046 +0000 UTC m=+1340.409457723" watchObservedRunningTime="2026-11-01 20:20:52.691651527 +0000 UTC m=+1340.412730204"
Nov 01 20:20:52.925846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:52.925648    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:20:54.137028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.136999    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:54.163815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.161277    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\") pod \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") "
Nov 01 20:20:54.163815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.161596    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\" (UniqueName: \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\") pod \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") "
Nov 01 20:20:54.163815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.161662    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\") pod \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\" (UID: \"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3\") "
Nov 01 20:20:54.182875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.182720    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca" pod "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" (UID: "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:20:54.183702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.183672    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:54.201844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.198344    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l" pod "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" (UID: "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3"). InnerVolumeSpecName "kube-api-access-d555l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:20:54.413803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.388329    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d555l\" (UniqueName: \"kubernetes.io/projected/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-kube-api-access-d555l\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:54.509378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.508057    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca" pod "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" (UID: "b264aa24-3a85-4f00-b6ce-024bb2ad1dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:20:54.532873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.528446    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:20:54.532873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.529465    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" containerName="revoke-keys"
Nov 01 20:20:54.532873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.529481    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" containerName="revoke-keys"
Nov 01 20:20:54.532873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.529629    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" containerName="revoke-keys"
Nov 01 20:20:54.538799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.537011    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.563804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.560734    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-036c69-openshift-default-dockercfg-9njcd\""
Nov 01 20:20:54.584797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.576841    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:20:54.588167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.585588    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:20:54.650816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.649701    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"
Nov 01 20:20:54.650816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.649733    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt" event={"ID":"b264aa24-3a85-4f00-b6ce-024bb2ad1dc3","Type":"ContainerDied","Data":"9e7711df81ea0499d17f5d283472b10ea60197cc341672d382a834d97d77337e"}
Nov 01 20:20:54.650816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.649794    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e7711df81ea0499d17f5d283472b10ea60197cc341672d382a834d97d77337e"
Nov 01 20:20:54.687337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.687300    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.687569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.687530    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85nlz\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.687709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.687681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.687934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.687907    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.688024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.687965    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.688094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.688075    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.688175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.688155    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.688260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.688244    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.688307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.688274    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790073    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85nlz\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790167    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790303    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790339    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790391    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790458    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790554    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.790992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.790596    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.791639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.791167    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.791639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.791415    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.791763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.791666    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.791858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.791795    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.791983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.791959    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.793548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.793526    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.794648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.794626    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.799617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.799590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85nlz\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.800012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.799991    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:54.887198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:54.887111    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:55.209225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.209128    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:20:55.209745 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:20:55.209716    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c276b11_b7a7_4717_b385_9e1c04b1820a.slice/crio-7175b65fb4dd746d93e0410fa4ba4b856ec7db709be46d5c1e89fecbbea85fa3 WatchSource:0}: Error finding container 7175b65fb4dd746d93e0410fa4ba4b856ec7db709be46d5c1e89fecbbea85fa3: Status 404 returned error can't find the container with id 7175b65fb4dd746d93e0410fa4ba4b856ec7db709be46d5c1e89fecbbea85fa3
Nov 01 20:20:55.212923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.212877    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:55.213036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.212971    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:55.213036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.213018    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:55.662597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.662543    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerStarted","Data":"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"}
Nov 01 20:20:55.662958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.662931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerStarted","Data":"7175b65fb4dd746d93e0410fa4ba4b856ec7db709be46d5c1e89fecbbea85fa3"}
Nov 01 20:20:55.687207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.687146    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podStartSLOduration=1.687126894 podStartE2EDuration="1.687126894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:55.680508078 +0000 UTC m=+1343.401586758" watchObservedRunningTime="2026-11-01 20:20:55.687126894 +0000 UTC m=+1343.408205573"
Nov 01 20:20:55.892176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:55.892144    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:56.900474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:56.900430    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:57.679661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:57.679631    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:57.680621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:57.680605    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:20:58.545526 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:20:58.545491    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:20:59.026451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.026374    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"]
Nov 01 20:20:59.035815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.034433    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e4d566c-866ss"]
Nov 01 20:20:59.698892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.698856    5041 generic.go:356] "Generic (PLEG): container finished" podID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerID="313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4" exitCode=0
Nov 01 20:20:59.699285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.698938    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerDied","Data":"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"}
Nov 01 20:20:59.699723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.699668    5041 scope.go:122] "RemoveContainer" containerID="313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"
Nov 01 20:20:59.701361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.701328    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:59.701483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.701409    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:59.701483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.701462    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:20:59.985952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.985866    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:20:59.989687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:20:59.989656    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.000089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.000055    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:21:00.000475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.000440    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"simulator-w1-a6873b-kserve-self-signed-certs\""
Nov 01 20:21:00.000626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.000552    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"default-dockercfg-vklkk\""
Nov 01 20:21:00.000626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.000557    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"kube-root-ca.crt\""
Nov 01 20:21:00.000761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.000628    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a6873b\"/\"openshift-service-ca.crt\""
Nov 01 20:21:00.044848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.044812    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.045058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.044957    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.045058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.045028    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.045188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.045141    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.045188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.045172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974h7\" (UniqueName: \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.045300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.045267    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.147566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.147503    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.148948    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.149017    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.149093    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.149122    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974h7\" (UniqueName: \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.149183    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.149658    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.148037    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.168454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.158338    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.193623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.193548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974h7\" (UniqueName: \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.207803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.195872    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.225799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.220978    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\") pod \"simulator-w1-a6873b-kserve-789478ff89-vddl5\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") " pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.300844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.299906    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"]
Nov 01 20:21:00.311138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.310059    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.311477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.311445    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"]
Nov 01 20:21:00.311585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.311551    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:00.316273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.316247    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:21:00.316352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.316280    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:21:00.316352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.316323    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:21:00.318832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.316738    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:21:00.363214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.363158    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.363493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.363469    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.363582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.363516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kp8b\" (UniqueName: \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.447832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.447325    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6903c179-0d1b-46fd-ab90-898741d8cd3a" path="/var/lib/kubelet/pods/6903c179-0d1b-46fd-ab90-898741d8cd3a/volumes"
Nov 01 20:21:00.469938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.466820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.469938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.466885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kp8b\" (UniqueName: \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.469938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.467086    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.469938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.468256    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.472166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.472135    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.476072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.476016    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kp8b\" (UniqueName: \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\") pod \"maas-api-revoke-sub-43bcb3ef-qc5hb\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.638045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.637975    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:00.653746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.653327    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:21:00.679475 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:00.679418    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa562803_46c5_4958_b192_133a03f0766f.slice/crio-df31f2b5b97d644a9b189cd17ad3589f3ee119700dbfefd51d03e813b28c2967 WatchSource:0}: Error finding container df31f2b5b97d644a9b189cd17ad3589f3ee119700dbfefd51d03e813b28c2967: Status 404 returned error can't find the container with id df31f2b5b97d644a9b189cd17ad3589f3ee119700dbfefd51d03e813b28c2967
Nov 01 20:21:00.710855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.710820    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" event={"ID":"aa562803-46c5-4958-b192-133a03f0766f","Type":"ContainerStarted","Data":"df31f2b5b97d644a9b189cd17ad3589f3ee119700dbfefd51d03e813b28c2967"}
Nov 01 20:21:00.713963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.713931    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerStarted","Data":"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"}
Nov 01 20:21:00.988853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:00.973911    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"]
Nov 01 20:21:01.088939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.088905    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:21:01.090425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.090398    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:01.090567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.090448    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:01.622091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.622040    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:21:01.728643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.728594    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" event={"ID":"aa562803-46c5-4958-b192-133a03f0766f","Type":"ContainerStarted","Data":"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"}
Nov 01 20:21:01.730750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.730680    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb" event={"ID":"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5","Type":"ContainerStarted","Data":"0f5598195250f91a04064848ef5803c660f1cec1096d9367f92d8968ee30d107"}
Nov 01 20:21:01.730750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.730730    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb" event={"ID":"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5","Type":"ContainerStarted","Data":"2ed85e9ffcd771175ec9caad19fb37d48c8a6a281c45522fe67023d295f4aa1a"}
Nov 01 20:21:01.769500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.769439    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" podStartSLOduration=2.76942338 podStartE2EDuration="2.76942338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:01.754178113 +0000 UTC m=+1349.475256803" watchObservedRunningTime="2026-11-01 20:21:01.76942338 +0000 UTC m=+1349.490502055"
Nov 01 20:21:01.778139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.778088    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb" podStartSLOduration=1.7780745260000002 podStartE2EDuration="1.778074526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:01.774317327 +0000 UTC m=+1349.495396004" watchObservedRunningTime="2026-11-01 20:21:01.778074526 +0000 UTC m=+1349.499153201"
Nov 01 20:21:01.924722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.924673    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:21:01.988090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:01.987989    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:21:02.089572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.089545    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:02.089706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.089597    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:02.312560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.312531    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:02.766861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.764095    5041 generic.go:356] "Generic (PLEG): container finished" podID="21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" containerID="0f5598195250f91a04064848ef5803c660f1cec1096d9367f92d8968ee30d107" exitCode=0
Nov 01 20:21:02.766861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.765339    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb" event={"ID":"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5","Type":"ContainerDied","Data":"0f5598195250f91a04064848ef5803c660f1cec1096d9367f92d8968ee30d107"}
Nov 01 20:21:02.816115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:02.816086    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:21:03.089112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:03.089078    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:03.089300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:03.089145    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:03.366961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:03.366870    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:21:04.021455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.021426    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:04.089669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.089633    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:04.090331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.090290    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:04.177713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.177678    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\" (UniqueName: \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\") pod \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") "
Nov 01 20:21:04.177713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.177719    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\") pod \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") "
Nov 01 20:21:04.178061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.178025    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\") pod \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\" (UID: \"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5\") "
Nov 01 20:21:04.178398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.178373    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca" pod "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" (UID: "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:04.178882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.178861    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:04.180431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.180409    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca" pod "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" (UID: "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:21:04.180511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.180442    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b" pod "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" (UID: "21ca2714-5d58-4cd0-836d-b9b8ccdd2be5"). InnerVolumeSpecName "kube-api-access-8kp8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:04.280630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.280599    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kp8b\" (UniqueName: \"kubernetes.io/projected/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-kube-api-access-8kp8b\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:04.280630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.280628    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:04.789688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.789018    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"
Nov 01 20:21:04.789688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.789172    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb" event={"ID":"21ca2714-5d58-4cd0-836d-b9b8ccdd2be5","Type":"ContainerDied","Data":"2ed85e9ffcd771175ec9caad19fb37d48c8a6a281c45522fe67023d295f4aa1a"}
Nov 01 20:21:04.789688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:04.789223    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ed85e9ffcd771175ec9caad19fb37d48c8a6a281c45522fe67023d295f4aa1a"
Nov 01 20:21:05.089459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:05.089378    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:05.089861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:05.089457    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:06.089271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:06.089235    5041 patch_prober.go:29] interesting pod/e2e-autores-933245-openshift-default-8b8b4856c-nbw24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Nov 01 20:21:06.089464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:06.089294    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Nov 01 20:21:06.901131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:06.901095    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:21:07.115061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:07.109075    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:21:07.121694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:07.121664    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:21:10.311838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:10.311798    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:10.324645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:10.324618    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:10.892729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:10.892682    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:21:12.559406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.559371    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:12.568153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.568096    5041 cpu_manager.go:414] "containerMap: removing container" podUID="21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" containerName="revoke-keys"
Nov 01 20:21:12.568153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.568134    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" containerName="revoke-keys"
Nov 01 20:21:12.568431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.568363    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" containerName="revoke-keys"
Nov 01 20:21:12.571934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.571906    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.577482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.577438    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:12.675297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.675256    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.675600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.675570    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754l2\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.676662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.676637    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.676833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.676812    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.778419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.778371    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.778640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.778433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754l2\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.778640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.778490    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.778640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.778524    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.779742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.779712    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.780161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.780141    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.783639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.783613    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.835844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.834321    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-754l2\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:12.900873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:12.899060    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:13.074273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.074239    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"]
Nov 01 20:21:13.091564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.088960    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"]
Nov 01 20:21:13.091564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.089054    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.095618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.095473    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:21:13.095618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.095538    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:21:13.095618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.095582    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:21:13.097108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.097051    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:21:13.218150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.218102    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.218306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.218248    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xcb\" (UniqueName: \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.218368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.218303    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.297449 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:13.297406    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f6c3fa_fa85_4df8_ac90_eac3ad554d39.slice/crio-80a3ab2d17bd4b61fd39c1fd9e3b10f005f94f0d616fa7395b22f823a067cbe6 WatchSource:0}: Error finding container 80a3ab2d17bd4b61fd39c1fd9e3b10f005f94f0d616fa7395b22f823a067cbe6: Status 404 returned error can't find the container with id 80a3ab2d17bd4b61fd39c1fd9e3b10f005f94f0d616fa7395b22f823a067cbe6
Nov 01 20:21:13.298556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.298525    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:13.319894 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.319866    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xcb\" (UniqueName: \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.320001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.319919    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.320244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.320223    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.320681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.320641    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.323036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.323014    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.328860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.328841    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xcb\" (UniqueName: \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\") pod \"maas-api-revoke-sub-361c20ca-9bhk9\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.425025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.424988    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:13.726159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:13.726133    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"]
Nov 01 20:21:13.726477 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:13.726133    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9d99d9_6158_4cb6_841d_1e445d588bf6.slice/crio-1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767 WatchSource:0}: Error finding container 1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767: Status 404 returned error can't find the container with id 1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767
Nov 01 20:21:14.029238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:14.029146    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" event={"ID":"63f6c3fa-fa85-4df8-ac90-eac3ad554d39","Type":"ContainerStarted","Data":"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"}
Nov 01 20:21:14.029238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:14.029205    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" event={"ID":"63f6c3fa-fa85-4df8-ac90-eac3ad554d39","Type":"ContainerStarted","Data":"80a3ab2d17bd4b61fd39c1fd9e3b10f005f94f0d616fa7395b22f823a067cbe6"}
Nov 01 20:21:14.029457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:14.029358    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:14.030657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:14.030626    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9" event={"ID":"bf9d99d9-6158-4cb6-841d-1e445d588bf6","Type":"ContainerStarted","Data":"1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767"}
Nov 01 20:21:14.048897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:14.048852    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" podStartSLOduration=2.048838854 podStartE2EDuration="2.048838854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:14.043501344 +0000 UTC m=+1361.764580020" watchObservedRunningTime="2026-11-01 20:21:14.048838854 +0000 UTC m=+1361.769917556"
Nov 01 20:21:15.040751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:15.040717    5041 generic.go:356] "Generic (PLEG): container finished" podID="bf9d99d9-6158-4cb6-841d-1e445d588bf6" containerID="cb0400f7e801eec571560f56a3640d66bf34b4474be3c77f6392e7be98a414b7" exitCode=0
Nov 01 20:21:15.041238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:15.040804    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9" event={"ID":"bf9d99d9-6158-4cb6-841d-1e445d588bf6","Type":"ContainerDied","Data":"cb0400f7e801eec571560f56a3640d66bf34b4474be3c77f6392e7be98a414b7"}
Nov 01 20:21:16.257170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.257138    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:16.368067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.367980    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\") pod \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") "
Nov 01 20:21:16.368342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.368317    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\" (UniqueName: \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\") pod \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") "
Nov 01 20:21:16.368419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.368360    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\") pod \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\" (UID: \"bf9d99d9-6158-4cb6-841d-1e445d588bf6\") "
Nov 01 20:21:16.368913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.368888    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca" pod "bf9d99d9-6158-4cb6-841d-1e445d588bf6" (UID: "bf9d99d9-6158-4cb6-841d-1e445d588bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:16.370838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.370773    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb" pod "bf9d99d9-6158-4cb6-841d-1e445d588bf6" (UID: "bf9d99d9-6158-4cb6-841d-1e445d588bf6"). InnerVolumeSpecName "kube-api-access-25xcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:16.371750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.371724    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca" pod "bf9d99d9-6158-4cb6-841d-1e445d588bf6" (UID: "bf9d99d9-6158-4cb6-841d-1e445d588bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:21:16.470106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.470075    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25xcb\" (UniqueName: \"kubernetes.io/projected/bf9d99d9-6158-4cb6-841d-1e445d588bf6-kube-api-access-25xcb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:16.470106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.470104    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:16.470308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:16.470113    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf9d99d9-6158-4cb6-841d-1e445d588bf6-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:17.072765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:17.071353    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"
Nov 01 20:21:17.072765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:17.071366    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9" event={"ID":"bf9d99d9-6158-4cb6-841d-1e445d588bf6","Type":"ContainerDied","Data":"1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767"}
Nov 01 20:21:17.072765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:17.071534    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc850d32d43d121b74b26c28bd2deb6ba33dbc5b87b511c89bbb71f30ddf767"
Nov 01 20:21:21.935801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:21.933881    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" podUID="33924fab-f553-4b02-b482-dd95d1c49f30" containerName="main" containerID="cri-o://e5340d73c5e03362e422e48a3463286608f136a38a3d868c4ec7b9fb5dd1390e" gracePeriod=45
Nov 01 20:21:22.118733 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.118702    5041 generic.go:356] "Generic (PLEG): container finished" podID="33924fab-f553-4b02-b482-dd95d1c49f30" containerID="e5340d73c5e03362e422e48a3463286608f136a38a3d868c4ec7b9fb5dd1390e" exitCode=0
Nov 01 20:21:22.118963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.118800    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" event={"ID":"33924fab-f553-4b02-b482-dd95d1c49f30","Type":"ContainerDied","Data":"e5340d73c5e03362e422e48a3463286608f136a38a3d868c4ec7b9fb5dd1390e"}
Nov 01 20:21:22.254813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.254764    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:21:22.353657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353626    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.353817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353685    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.353817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353786    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.353923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353816    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.353923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353862    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\" (UniqueName: \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.354037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353936    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:22.354037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.353995    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\" (UniqueName: \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\") pod \"33924fab-f553-4b02-b482-dd95d1c49f30\" (UID: \"33924fab-f553-4b02-b482-dd95d1c49f30\") "
Nov 01 20:21:22.354136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.354109    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:22.354189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.354140    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:22.354674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.354651    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:22.354845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.354688    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:22.354845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.354698    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:22.356287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.356259    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:22.356403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.356286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:21:22.356403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.356325    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf" pod "33924fab-f553-4b02-b482-dd95d1c49f30" (UID: "33924fab-f553-4b02-b482-dd95d1c49f30"). InnerVolumeSpecName "kube-api-access-p8nmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:22.456217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.456186    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33924fab-f553-4b02-b482-dd95d1c49f30-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:22.456217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.456214    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8nmf\" (UniqueName: \"kubernetes.io/projected/33924fab-f553-4b02-b482-dd95d1c49f30-kube-api-access-p8nmf\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:22.456217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:22.456225    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33924fab-f553-4b02-b482-dd95d1c49f30-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:23.128612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:23.128579    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526" event={"ID":"33924fab-f553-4b02-b482-dd95d1c49f30","Type":"ContainerDied","Data":"d4e90ce472ff4ec25b6caac8e0823f373db008c3d7a42a9adbd0b18bbd61dcca"}
Nov 01 20:21:23.129060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:23.128624    5041 scope.go:122] "RemoveContainer" containerID="e5340d73c5e03362e422e48a3463286608f136a38a3d868c4ec7b9fb5dd1390e"
Nov 01 20:21:23.129060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:23.128586    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"
Nov 01 20:21:23.148067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:23.148037    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:21:23.152565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:23.152538    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-933245/autores-model-933245-kserve-7c7f8468f7-62526"]
Nov 01 20:21:24.432201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:24.432063    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33924fab-f553-4b02-b482-dd95d1c49f30" path="/var/lib/kubelet/pods/33924fab-f553-4b02-b482-dd95d1c49f30/volumes"
Nov 01 20:21:25.081804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.075018    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:25.114804 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.114091    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.114804 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.114124    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.114804 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.114189    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:25.614168963 +0000 UTC m=+1373.335247634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.193800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.170505    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:21:25.623549 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.623518    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"]
Nov 01 20:21:25.625172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.625150    5041 cpu_manager.go:414] "containerMap: removing container" podUID="33924fab-f553-4b02-b482-dd95d1c49f30" containerName="main"
Nov 01 20:21:25.625323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.625309    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33924fab-f553-4b02-b482-dd95d1c49f30" containerName="main"
Nov 01 20:21:25.625448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.625438    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bf9d99d9-6158-4cb6-841d-1e445d588bf6" containerName="revoke-keys"
Nov 01 20:21:25.626126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.626107    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9d99d9-6158-4cb6-841d-1e445d588bf6" containerName="revoke-keys"
Nov 01 20:21:25.626411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.626399    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9d99d9-6158-4cb6-841d-1e445d588bf6" containerName="revoke-keys"
Nov 01 20:21:25.626514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.626504    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="33924fab-f553-4b02-b482-dd95d1c49f30" containerName="main"
Nov 01 20:21:25.634059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.634040    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"]
Nov 01 20:21:25.634221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.634154    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.640372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.640048    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:21:25.641127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.641077    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:21:25.642006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.641980    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:21:25.642818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.642718    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:21:25.671651 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.671621    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.671793 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.671656    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.671793 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:25.671754    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:26.671733398 +0000 UTC m=+1374.392812070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : secret "agc-cred-8876b9054304" not found
Nov 01 20:21:25.772663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.772632    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkxz\" (UniqueName: \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.772859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.772672    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.772927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.772907    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.874100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.874027    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkxz\" (UniqueName: \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.874100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.874074    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.874337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.874227    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.874949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.874919    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.876871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.876836    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.883954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.883928    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkxz\" (UniqueName: \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\") pod \"maas-api-revoke-sub-b91dd056-s5cqz\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:25.953034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:25.953006    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:26.239346 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:26.239317    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc11cd7_5bd4_4a41_8db2_07e1c203ad91.slice/crio-ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf WatchSource:0}: Error finding container ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf: Status 404 returned error can't find the container with id ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf
Nov 01 20:21:26.241087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:26.241059    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"]
Nov 01 20:21:26.687229 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:26.687200    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:26.687229 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:26.687231    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:26.687620 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:26.687298    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:28.687284697 +0000 UTC m=+1376.408363351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : secret "agc-cred-8876b9054304" not found
Nov 01 20:21:27.222136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:27.222105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz" event={"ID":"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91","Type":"ContainerStarted","Data":"9bc7632f3ef6dba346b5d0d219935c945977b4ae1972065656ef4e8ce31c277c"}
Nov 01 20:21:27.222314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:27.222156    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz" event={"ID":"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91","Type":"ContainerStarted","Data":"ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf"}
Nov 01 20:21:27.240495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:27.240427    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz" podStartSLOduration=2.240410628 podStartE2EDuration="2.240410628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:27.237555955 +0000 UTC m=+1374.958634632" watchObservedRunningTime="2026-11-01 20:21:27.240410628 +0000 UTC m=+1374.961489304"
Nov 01 20:21:28.233330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:28.233297    5041 generic.go:356] "Generic (PLEG): container finished" podID="0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" containerID="9bc7632f3ef6dba346b5d0d219935c945977b4ae1972065656ef4e8ce31c277c" exitCode=0
Nov 01 20:21:28.233849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:28.233390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz" event={"ID":"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91","Type":"ContainerDied","Data":"9bc7632f3ef6dba346b5d0d219935c945977b4ae1972065656ef4e8ce31c277c"}
Nov 01 20:21:28.543713 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:28.543681    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:21:28.718316 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:28.718286    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:28.719773 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:28.718736    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:28.719773 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:28.718859    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:32.7188367 +0000 UTC m=+1380.439915368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : secret "agc-cred-8876b9054304" not found
Nov 01 20:21:29.427312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.427281    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:29.531701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.531624    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\") pod \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") "
Nov 01 20:21:29.531921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.531903    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\" (UniqueName: \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\") pod \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") "
Nov 01 20:21:29.531976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.531934    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\") pod \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\" (UID: \"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91\") "
Nov 01 20:21:29.532018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.531995    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca" pod "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" (UID: "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:29.532434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.532417    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:29.534078 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.534050    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz" pod "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" (UID: "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91"). InnerVolumeSpecName "kube-api-access-pmkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:29.534184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.534150    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca" pod "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" (UID: "0dc11cd7-5bd4-4a41-8db2-07e1c203ad91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:21:29.633637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.633599    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:29.633637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:29.633632    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkxz\" (UniqueName: \"kubernetes.io/projected/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91-kube-api-access-pmkxz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:30.172336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.172219    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" podUID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" containerName="payload-processing" containerID="cri-o://b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc" gracePeriod=25
Nov 01 20:21:30.252681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.252644    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz" event={"ID":"0dc11cd7-5bd4-4a41-8db2-07e1c203ad91","Type":"ContainerDied","Data":"ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf"}
Nov 01 20:21:30.252681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.252666    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"
Nov 01 20:21:30.252939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.252732    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea856664370a8e128cfce92d17e6094961780aff4008fb45102cf1b47ca277cf"
Nov 01 20:21:30.504757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.504734    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:21:30.648726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.648694    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\") pod \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") "
Nov 01 20:21:30.649031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.648772    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\") pod \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") "
Nov 01 20:21:30.649031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.648843    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\") pod \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") "
Nov 01 20:21:30.649031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.648895    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\") pod \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\" (UID: \"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2\") "
Nov 01 20:21:30.649287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.649260    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume" pod "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" (UID: "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:30.649853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.649827    5041 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-routing-overlay-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:30.650225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.650198    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume" pod "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" (UID: "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:30.651274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.651248    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials" pod "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" (UID: "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:30.651358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.651280    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj" pod "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" (UID: "7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2"). InnerVolumeSpecName "kube-api-access-27csj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:30.751207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.751132    5041 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-provider-credentials\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:30.751207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.751161    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:30.751207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:30.751172    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27csj\" (UniqueName: \"kubernetes.io/projected/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2-kube-api-access-27csj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:31.057743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.057709    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-34b933"]
Nov 01 20:21:31.058491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058471    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" containerName="payload-processing"
Nov 01 20:21:31.058552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058495    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" containerName="payload-processing"
Nov 01 20:21:31.058552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058543    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" containerName="revoke-keys"
Nov 01 20:21:31.058632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058552    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" containerName="revoke-keys"
Nov 01 20:21:31.058753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058739    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" containerName="payload-processing"
Nov 01 20:21:31.067529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.058766    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" containerName="revoke-keys"
Nov 01 20:21:31.072134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.072108    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-34b933"]
Nov 01 20:21:31.072306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.072199    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:31.076152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.076128    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\""
Nov 01 20:21:31.158402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.158368    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc7c\" (UniqueName: \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\") pod \"test-curl-7426-34b933\" (UID: \"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e\") " pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:31.260068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.260033    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc7c\" (UniqueName: \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\") pod \"test-curl-7426-34b933\" (UID: \"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e\") " pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:31.263697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.263670    5041 generic.go:356] "Generic (PLEG): container finished" podID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" containerID="b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc" exitCode=0
Nov 01 20:21:31.263845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.263722    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"
Nov 01 20:21:31.263845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.263761    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" event={"ID":"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2","Type":"ContainerDied","Data":"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"}
Nov 01 20:21:31.263845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.263841    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz" event={"ID":"7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2","Type":"ContainerDied","Data":"c6f9afe70fb123f9ef35d0481840e21149b50e5237ecc7df4872e9d15ea8132d"}
Nov 01 20:21:31.264018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.263901    5041 scope.go:122] "RemoveContainer" containerID="b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"
Nov 01 20:21:31.268974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.268953    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc7c\" (UniqueName: \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\") pod \"test-curl-7426-34b933\" (UID: \"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e\") " pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:31.290928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.290717    5041 scope.go:122] "RemoveContainer" containerID="b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"
Nov 01 20:21:31.291103 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:31.291080    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc\": container with ID starting with b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc not found: ID does not exist" containerID="b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"
Nov 01 20:21:31.291156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.291111    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc"} err="failed to get container status \"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc\": rpc error: code = NotFound desc = could not find container \"b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc\": container with ID starting with b4b35a2cb159666158e19b45ed0777aa6ca63390b5b3807051baa24f0f28e2fc not found: ID does not exist"
Nov 01 20:21:31.302980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.302957    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:21:31.307437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.307414    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-7bd5t7kvz"]
Nov 01 20:21:31.389026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.388944    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:31.672847 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:31.672816    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb1aa5f_45d4_4fd1_83a3_e9d3e110578e.slice/crio-fba38c4f4f53512f28bae12b1a61de5da0addb0a1e415f6fa6adbfe456150f7c WatchSource:0}: Error finding container fba38c4f4f53512f28bae12b1a61de5da0addb0a1e415f6fa6adbfe456150f7c: Status 404 returned error can't find the container with id fba38c4f4f53512f28bae12b1a61de5da0addb0a1e415f6fa6adbfe456150f7c
Nov 01 20:21:31.673360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.673336    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-34b933"]
Nov 01 20:21:31.675199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:31.675179    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:21:32.272727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:32.272685    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-34b933" event={"ID":"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e","Type":"ContainerStarted","Data":"fba38c4f4f53512f28bae12b1a61de5da0addb0a1e415f6fa6adbfe456150f7c"}
Nov 01 20:21:32.443209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:32.443179    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2" path="/var/lib/kubelet/pods/7ea0a61b-bbdf-43fa-9c98-2a6ea272dfc2/volumes"
Nov 01 20:21:32.799460 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:32.799430    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:32.799460 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:32.799459    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:32.800010 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:32.799521    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:40.799503751 +0000 UTC m=+1388.520582413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : secret "agc-cred-8876b9054304" not found
Nov 01 20:21:33.284328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:33.284231    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-34b933" event={"ID":"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e","Type":"ContainerStarted","Data":"9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1"}
Nov 01 20:21:33.298106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:33.298058    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-34b933" podStartSLOduration=1.006716822 podStartE2EDuration="2.29804405s" totalImagesPullingTime="1.291327228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:33.296602418 +0000 UTC m=+1381.017681095" watchObservedRunningTime="2026-11-01 20:21:33.29804405 +0000 UTC m=+1381.019122755"
Nov 01 20:21:33.941951 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:33.941906    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-34b933"]
Nov 01 20:21:33.947209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:33.947176    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-34b933"]
Nov 01 20:21:34.086511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.086468    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-685ca3"]
Nov 01 20:21:34.087502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.087477    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" containerName="test-curl-7426-34b933"
Nov 01 20:21:34.087502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.087502    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" containerName="test-curl-7426-34b933"
Nov 01 20:21:34.087739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.087723    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" containerName="test-curl-7426-34b933"
Nov 01 20:21:34.092262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.092239    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:34.093081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.093062    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-685ca3"]
Nov 01 20:21:34.224863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.224748    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnb9m\" (UniqueName: \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\") pod \"test-curl-7426-685ca3\" (UID: \"f82507b9-87ea-42aa-a861-f7b3d2cd6c38\") " pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:34.326352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.326324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnb9m\" (UniqueName: \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\") pod \"test-curl-7426-685ca3\" (UID: \"f82507b9-87ea-42aa-a861-f7b3d2cd6c38\") " pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:34.334235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.334208    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnb9m\" (UniqueName: \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\") pod \"test-curl-7426-685ca3\" (UID: \"f82507b9-87ea-42aa-a861-f7b3d2cd6c38\") " pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:34.409020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.408990    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:34.696117 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:34.696082    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf82507b9_87ea_42aa_a861_f7b3d2cd6c38.slice/crio-b42f12996a9d010324accfabc1de067248b39dd1b81192c313bc531e3f63a189 WatchSource:0}: Error finding container b42f12996a9d010324accfabc1de067248b39dd1b81192c313bc531e3f63a189: Status 404 returned error can't find the container with id b42f12996a9d010324accfabc1de067248b39dd1b81192c313bc531e3f63a189
Nov 01 20:21:34.696263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:34.696166    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-685ca3"]
Nov 01 20:21:35.268457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.268412    5041 scope.go:122] "RemoveContainer" containerID="6eb2f28711c55f36c8384d6849f492f0dee145a9e59d414fe7366b9c6b382fa0"
Nov 01 20:21:35.279969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.279945    5041 scope.go:122] "RemoveContainer" containerID="9903b7d31b9ae4f4dbfeb1ead972930bb15b996c4ae260e36cf9c70937701971"
Nov 01 20:21:35.290387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.290365    5041 scope.go:122] "RemoveContainer" containerID="3d0cee4f0179816be04f787f47c905dbe8d5ae8a52178215d618bb60c80f6d25"
Nov 01 20:21:35.301222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.301202    5041 scope.go:122] "RemoveContainer" containerID="97d446a82b58cc3f2285c5d465fe5206d449843655c143ce9481b6eb3d16003b"
Nov 01 20:21:35.301355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.301325    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-685ca3" event={"ID":"f82507b9-87ea-42aa-a861-f7b3d2cd6c38","Type":"ContainerStarted","Data":"ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1"}
Nov 01 20:21:35.301407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.301374    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-685ca3" event={"ID":"f82507b9-87ea-42aa-a861-f7b3d2cd6c38","Type":"ContainerStarted","Data":"b42f12996a9d010324accfabc1de067248b39dd1b81192c313bc531e3f63a189"}
Nov 01 20:21:35.303993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.303949    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-34b933" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" containerName="test-curl-7426-34b933" containerID="cri-o://9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1" gracePeriod=2
Nov 01 20:21:35.318688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:35.318629    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-685ca3" podStartSLOduration=1.318609995 podStartE2EDuration="1.318609995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:35.315393227 +0000 UTC m=+1383.036471905" watchObservedRunningTime="2026-11-01 20:21:35.318609995 +0000 UTC m=+1383.039688676"
Nov 01 20:21:36.272853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.257599    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-685ca3"]
Nov 01 20:21:36.300215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.300179    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-685ca3"]
Nov 01 20:21:36.312149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.308819    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7426-f8e37f"]
Nov 01 20:21:36.312149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.309805    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" containerName="test-curl-7426-685ca3"
Nov 01 20:21:36.312149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.309826    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" containerName="test-curl-7426-685ca3"
Nov 01 20:21:36.312149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.310055    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" containerName="test-curl-7426-685ca3"
Nov 01 20:21:36.318256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.318232    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-f8e37f"]
Nov 01 20:21:36.318409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.318334    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:36.419612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.419578    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmc6z\" (UniqueName: \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\") pod \"test-curl-7426-f8e37f\" (UID: \"e2d97ae0-93da-476b-8e17-75109cb29789\") " pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:36.521033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.520990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmc6z\" (UniqueName: \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\") pod \"test-curl-7426-f8e37f\" (UID: \"e2d97ae0-93da-476b-8e17-75109cb29789\") " pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:36.529992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.529914    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmc6z\" (UniqueName: \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\") pod \"test-curl-7426-f8e37f\" (UID: \"e2d97ae0-93da-476b-8e17-75109cb29789\") " pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:36.653800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.653027    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:36.877994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:36.877959    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:37.193202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:37.191504    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7426-f8e37f"]
Nov 01 20:21:37.720697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:37.720071    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-685ca3" podUID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" containerName="test-curl-7426-685ca3" containerID="cri-o://ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1" gracePeriod=2
Nov 01 20:21:37.720697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:37.720463    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-f8e37f" event={"ID":"e2d97ae0-93da-476b-8e17-75109cb29789","Type":"ContainerStarted","Data":"1dcafa70f74bdf0d51c832d32e45591c56b591afbb34911d9bde6c379866f8e8"}
Nov 01 20:21:38.599218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.599013    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:38.732429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.732397    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7426-f8e37f" event={"ID":"e2d97ae0-93da-476b-8e17-75109cb29789","Type":"ContainerStarted","Data":"68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2"}
Nov 01 20:21:38.734794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.734741    5041 generic.go:356] "Generic (PLEG): container finished" podID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" containerID="9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1" exitCode=137
Nov 01 20:21:38.734932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.734854    5041 scope.go:122] "RemoveContainer" containerID="9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1"
Nov 01 20:21:38.735174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.735052    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:38.735331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.735314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\" (UniqueName: \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\") pod \"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e\" (UID: \"0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e\") "
Nov 01 20:21:38.739768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.739745    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c" pod "0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" (UID: "0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e"). InnerVolumeSpecName "kube-api-access-hhc7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:38.780295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.780229    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7426-f8e37f" podStartSLOduration=2.780205011 podStartE2EDuration="2.780205011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:38.772136712 +0000 UTC m=+1386.493215392" watchObservedRunningTime="2026-11-01 20:21:38.780205011 +0000 UTC m=+1386.501283691"
Nov 01 20:21:38.790883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.790856    5041 scope.go:122] "RemoveContainer" containerID="9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1"
Nov 01 20:21:38.791314 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:38.791285    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1\": container with ID starting with 9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1 not found: ID does not exist" containerID="9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1"
Nov 01 20:21:38.791396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.791331    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1"} err="failed to get container status \"9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1\": rpc error: code = NotFound desc = could not find container \"9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1\": container with ID starting with 9abda98ea1ad2f79900f34e876f9a865dd55eac00180cf7d575aea6f5683edf1 not found: ID does not exist"
Nov 01 20:21:38.838322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:38.838240    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhc7c\" (UniqueName: \"kubernetes.io/projected/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e-kube-api-access-hhc7c\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:39.065862 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:39.061956    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-34b933\" is forbidden: User \"system:node:ip-10-0-137-175.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-175.us-east-2.compute.internal' and this object" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:39.428023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:39.427977    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7426-f8e37f"]
Nov 01 20:21:39.435184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:39.435138    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7426-f8e37f"]
Nov 01 20:21:40.107437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.107405    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:21:40.109865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.109839    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" containerName="test-curl-7426-f8e37f"
Nov 01 20:21:40.110021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.110006    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" containerName="test-curl-7426-f8e37f"
Nov 01 20:21:40.110367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.110346    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" containerName="test-curl-7426-f8e37f"
Nov 01 20:21:40.134961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.130364    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.134961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.130656    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:21:40.140455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.140428    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"default-dockercfg-5zn99\""
Nov 01 20:21:40.163800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.140805    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"rate-test-model-8b6cd6-kserve-self-signed-certs\""
Nov 01 20:21:40.163800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.140931    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"openshift-service-ca.crt\""
Nov 01 20:21:40.163800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.141023    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"kube-root-ca.crt\""
Nov 01 20:21:40.163800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.149958    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:40.179715 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.179685    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-34b933\" is forbidden: User \"system:node:ip-10-0-137-175.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-175.us-east-2.compute.internal' and this object" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:40.183031 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.182998    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-34b933\" is forbidden: User \"system:node:ip-10-0-137-175.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-175.us-east-2.compute.internal' and this object" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" pod="opendatahub/test-curl-7426-34b933"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.305285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\" (UniqueName: \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\") pod \"f82507b9-87ea-42aa-a861-f7b3d2cd6c38\" (UID: \"f82507b9-87ea-42aa-a861-f7b3d2cd6c38\") "
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.305925    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.306050    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.306119    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jsz\" (UniqueName: \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.306175    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.306248    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.309864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.306393    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.331802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.328329    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m" pod "f82507b9-87ea-42aa-a861-f7b3d2cd6c38" (UID: "f82507b9-87ea-42aa-a861-f7b3d2cd6c38"). InnerVolumeSpecName "kube-api-access-rnb9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413238    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jsz\" (UniqueName: \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413388    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413456    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413574    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413662    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.413842    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnb9m\" (UniqueName: \"kubernetes.io/projected/f82507b9-87ea-42aa-a861-f7b3d2cd6c38-kube-api-access-rnb9m\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.414285    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.414550    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.416812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.416102    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.450800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.449722    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.467810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.467077    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.470813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.469445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jsz\" (UniqueName: \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\") pod \"rate-test-model-8b6cd6-kserve-559f954b85-7gzkl\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.479072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.479039    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e" path="/var/lib/kubelet/pods/0cb1aa5f-45d4-4fd1-83a3-e9d3e110578e/volumes"
Nov 01 20:21:40.479648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.479628    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" path="/var/lib/kubelet/pods/f82507b9-87ea-42aa-a861-f7b3d2cd6c38/volumes"
Nov 01 20:21:40.484192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.483232    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:40.791840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.791805    5041 generic.go:356] "Generic (PLEG): container finished" podID="f82507b9-87ea-42aa-a861-f7b3d2cd6c38" containerID="ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1" exitCode=137
Nov 01 20:21:40.792002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.791900    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-685ca3"
Nov 01 20:21:40.792002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.791957    5041 scope.go:122] "RemoveContainer" containerID="ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1"
Nov 01 20:21:40.792102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.792072    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7426-f8e37f" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" containerName="test-curl-7426-f8e37f" containerID="cri-o://68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2" gracePeriod=2
Nov 01 20:21:40.801900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.801878    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:21:40.814172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.814148    5041 scope.go:122] "RemoveContainer" containerID="ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1"
Nov 01 20:21:40.814574 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.814546    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1\": container with ID starting with ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1 not found: ID does not exist" containerID="ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1"
Nov 01 20:21:40.814642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:40.814587    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1"} err="failed to get container status \"ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1\": rpc error: code = NotFound desc = could not find container \"ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1\": container with ID starting with ae0ac86d723bbfd870129e045e39a6d514108b8e024a14eb7159143a522487e1 not found: ID does not exist"
Nov 01 20:21:40.816907 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:40.816859    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ae79db_b17d_4b0b_8756_316611154e4f.slice/crio-0236bb6f451456e22e1929b2e0bcf06511e1a325acb55b07c4d33b3859162b79 WatchSource:0}: Error finding container 0236bb6f451456e22e1929b2e0bcf06511e1a325acb55b07c4d33b3859162b79: Status 404 returned error can't find the container with id 0236bb6f451456e22e1929b2e0bcf06511e1a325acb55b07c4d33b3859162b79
Nov 01 20:21:40.819517 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.819496    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-8876b9054304: secret "agc-cred-8876b9054304" not found
Nov 01 20:21:40.819584 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.819520    5041 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/e2e-cfl-e2318b-provider-cred: secret "e2e-cfl-e2318b-provider-cred" not found
Nov 01 20:21:40.819584 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.819534    5041 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs: [secret "agc-cred-8876b9054304" not found, secret "e2e-cfl-e2318b-provider-cred" not found]
Nov 01 20:21:40.819657 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:40.819587    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials podName:63f6c3fa-fa85-4df8-ac90-eac3ad554d39 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:56.819566548 +0000 UTC m=+1404.540645204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39") : [secret "agc-cred-8876b9054304" not found, secret "e2e-cfl-e2318b-provider-cred" not found]
Nov 01 20:21:41.803677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:41.803637    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" event={"ID":"39ae79db-b17d-4b0b-8756-316611154e4f","Type":"ContainerStarted","Data":"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"}
Nov 01 20:21:41.804213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:41.803689    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" event={"ID":"39ae79db-b17d-4b0b-8756-316611154e4f","Type":"ContainerStarted","Data":"0236bb6f451456e22e1929b2e0bcf06511e1a325acb55b07c4d33b3859162b79"}
Nov 01 20:21:41.830960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:41.830869    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" podStartSLOduration=1.830851277 podStartE2EDuration="1.830851277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:41.820225967 +0000 UTC m=+1389.541304656" watchObservedRunningTime="2026-11-01 20:21:41.830851277 +0000 UTC m=+1389.551929953"
Nov 01 20:21:41.880707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:41.880636    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" podUID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" containerName="payload-processing" containerID="cri-o://72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134" gracePeriod=25
Nov 01 20:21:42.276804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.265581    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:42.382083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.381697    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\") pod \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") "
Nov 01 20:21:42.382083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.381770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\") pod \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") "
Nov 01 20:21:42.382083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.381878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\") pod \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") "
Nov 01 20:21:42.382083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.381950    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\") pod \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\" (UID: \"63f6c3fa-fa85-4df8-ac90-eac3ad554d39\") "
Nov 01 20:21:42.384254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.383799    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume" pod "63f6c3fa-fa85-4df8-ac90-eac3ad554d39" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:42.384254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.383947    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume" pod "63f6c3fa-fa85-4df8-ac90-eac3ad554d39" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:42.391364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.390957    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials" pod "63f6c3fa-fa85-4df8-ac90-eac3ad554d39" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:42.392587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.392556    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2" pod "63f6c3fa-fa85-4df8-ac90-eac3ad554d39" (UID: "63f6c3fa-fa85-4df8-ac90-eac3ad554d39"). InnerVolumeSpecName "kube-api-access-754l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:42.478251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.478215    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:21:42.478524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.478468    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy" containerID="cri-o://9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736" gracePeriod=30
Nov 01 20:21:42.484138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.484080    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-754l2\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-kube-api-access-754l2\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:42.484138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.484103    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:42.484138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.484112    5041 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-provider-credentials\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:42.484138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.484121    5041 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63f6c3fa-fa85-4df8-ac90-eac3ad554d39-routing-overlay-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:42.487514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.487463    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:42.814563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.814535    5041 generic.go:356] "Generic (PLEG): container finished" podID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" containerID="72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134" exitCode=0
Nov 01 20:21:42.814926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.814619    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"
Nov 01 20:21:42.815030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.815007    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" event={"ID":"63f6c3fa-fa85-4df8-ac90-eac3ad554d39","Type":"ContainerDied","Data":"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"}
Nov 01 20:21:42.815151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.815134    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs" event={"ID":"63f6c3fa-fa85-4df8-ac90-eac3ad554d39","Type":"ContainerDied","Data":"80a3ab2d17bd4b61fd39c1fd9e3b10f005f94f0d616fa7395b22f823a067cbe6"}
Nov 01 20:21:42.815246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.815170    5041 scope.go:122] "RemoveContainer" containerID="72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"
Nov 01 20:21:42.882732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.882693    5041 scope.go:122] "RemoveContainer" containerID="72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"
Nov 01 20:21:42.883132 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:42.883109    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134\": container with ID starting with 72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134 not found: ID does not exist" containerID="72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"
Nov 01 20:21:42.883195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.883145    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134"} err="failed to get container status \"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134\": rpc error: code = NotFound desc = could not find container \"72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134\": container with ID starting with 72fc35636c6e3222aa5230af99e71108951267d9f0e0b5a9f46411ef34ead134 not found: ID does not exist"
Nov 01 20:21:42.893333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.893308    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:42.897865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:42.897842    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-5745pzhhs"]
Nov 01 20:21:43.105451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.105420    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:43.194148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.194106    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\" (UniqueName: \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\") pod \"e2d97ae0-93da-476b-8e17-75109cb29789\" (UID: \"e2d97ae0-93da-476b-8e17-75109cb29789\") "
Nov 01 20:21:43.196330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.196286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z" pod "e2d97ae0-93da-476b-8e17-75109cb29789" (UID: "e2d97ae0-93da-476b-8e17-75109cb29789"). InnerVolumeSpecName "kube-api-access-zmc6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:43.295512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.295473    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmc6z\" (UniqueName: \"kubernetes.io/projected/e2d97ae0-93da-476b-8e17-75109cb29789-kube-api-access-zmc6z\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:43.825751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.825714    5041 generic.go:356] "Generic (PLEG): container finished" podID="e2d97ae0-93da-476b-8e17-75109cb29789" containerID="68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2" exitCode=137
Nov 01 20:21:43.826235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.825762    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:43.826235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.825853    5041 scope.go:122] "RemoveContainer" containerID="68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2"
Nov 01 20:21:43.838578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.838558    5041 scope.go:122] "RemoveContainer" containerID="68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2"
Nov 01 20:21:43.838928 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:43.838900    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2\": container with ID starting with 68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2 not found: ID does not exist" containerID="68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2"
Nov 01 20:21:43.839022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:43.838937    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2"} err="failed to get container status \"68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2\": rpc error: code = NotFound desc = could not find container \"68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2\": container with ID starting with 68de3435dec57af5648543deff1a1c771e976a757a0a79c7ff46932b513ccce2 not found: ID does not exist"
Nov 01 20:21:43.839934 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:43.839910    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-f8e37f\" is forbidden: User \"system:node:ip-10-0-137-175.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-175.us-east-2.compute.internal' and this object" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:44.384434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:44.384398    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:21:44.384608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:44.384588    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy" containerID="cri-o://81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367" gracePeriod=30
Nov 01 20:21:44.386847 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:44.386769    5041 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7426-f8e37f\" is forbidden: User \"system:node:ip-10-0-137-175.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-175.us-east-2.compute.internal' and this object" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" pod="opendatahub/test-curl-7426-f8e37f"
Nov 01 20:21:44.436279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:44.436240    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" path="/var/lib/kubelet/pods/63f6c3fa-fa85-4df8-ac90-eac3ad554d39/volumes"
Nov 01 20:21:44.436914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:44.436888    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d97ae0-93da-476b-8e17-75109cb29789" path="/var/lib/kubelet/pods/e2d97ae0-93da-476b-8e17-75109cb29789/volumes"
Nov 01 20:21:47.783617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.783554    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:21:47.877869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.877836    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.877923    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878068    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878162    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878262    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878318    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878413    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:47.878643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878440    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878550    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878592    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\") pod \"0867667c-ce38-4041-8ee5-a935a60d9401\" (UID: \"0867667c-ce38-4041-8ee5-a935a60d9401\") "
Nov 01 20:21:47.878643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878633    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:47.878914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878694    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:47.878970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.878952    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:47.879203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879168    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:47.879451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879434    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.879559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879454    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0867667c-ce38-4041-8ee5-a935a60d9401-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.879559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879470    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.879559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879483    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.879559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.879497    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.880809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.880760    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:21:47.881060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.881040    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:47.881138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.881122    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:47.882412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.882390    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n" pod "0867667c-ce38-4041-8ee5-a935a60d9401" (UID: "0867667c-ce38-4041-8ee5-a935a60d9401"). InnerVolumeSpecName "kube-api-access-jft8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:47.940537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.940496    5041 generic.go:356] "Generic (PLEG): container finished" podID="0867667c-ce38-4041-8ee5-a935a60d9401" containerID="9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736" exitCode=0
Nov 01 20:21:47.940735 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.940548    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"
Nov 01 20:21:47.940735 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.940579    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" event={"ID":"0867667c-ce38-4041-8ee5-a935a60d9401","Type":"ContainerDied","Data":"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"}
Nov 01 20:21:47.940735 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.940634    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk" event={"ID":"0867667c-ce38-4041-8ee5-a935a60d9401","Type":"ContainerDied","Data":"742657103c432d1f7ac32c4dbe7492bb05b9d65fb3a34ea006cecaf533181f04"}
Nov 01 20:21:47.940735 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.940660    5041 scope.go:122] "RemoveContainer" containerID="9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"
Nov 01 20:21:47.943393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.943366    5041 generic.go:356] "Generic (PLEG): container finished" podID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerID="05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a" exitCode=0
Nov 01 20:21:47.943514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.943442    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerDied","Data":"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"}
Nov 01 20:21:47.944034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.944012    5041 scope.go:122] "RemoveContainer" containerID="05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"
Nov 01 20:21:47.946835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.946800    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:47.946937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.946891    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:47.947003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.946940    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:47.954110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.954087    5041 scope.go:122] "RemoveContainer" containerID="9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"
Nov 01 20:21:47.954377 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:47.954358    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736\": container with ID starting with 9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736 not found: ID does not exist" containerID="9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"
Nov 01 20:21:47.954436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.954385    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736"} err="failed to get container status \"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736\": rpc error: code = NotFound desc = could not find container \"9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736\": container with ID starting with 9a9420f3b4d2b0766063764d09df94bd856e3fb2add4eb690cadd627971a5736 not found: ID does not exist"
Nov 01 20:21:47.980640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.980611    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jft8n\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-kube-api-access-jft8n\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.980640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.980638    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0867667c-ce38-4041-8ee5-a935a60d9401-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.980876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.980651    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0867667c-ce38-4041-8ee5-a935a60d9401-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.980876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.980660    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0867667c-ce38-4041-8ee5-a935a60d9401-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:47.984832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.984766    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:21:47.989959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:47.989905    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q5wnk"]
Nov 01 20:21:48.430605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:48.430571    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" path="/var/lib/kubelet/pods/0867667c-ce38-4041-8ee5-a935a60d9401/volumes"
Nov 01 20:21:48.985426 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:48.985391    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerStarted","Data":"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"}
Nov 01 20:21:49.865108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.865073    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:21:49.909804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.909304    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:21:49.909804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.909449    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:49.909804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.909506    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.943693    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.943759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.943829    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.943880    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.944060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.944105    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.944151    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.944182    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.944230    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\") pod \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\" (UID: \"bddffed5-308a-461f-bd60-1ce63aa6b4ba\") "
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.956407    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.957048    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.957246    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.962035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.957429    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:49.963215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:49.963186    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:21:50.057097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.054582    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.057097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.054613    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.057097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.054627    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.057097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.054641    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.057097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.054655    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.126367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.126084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:21:50.126367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.126323    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:50.128018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.127988    5041 generic.go:356] "Generic (PLEG): container finished" podID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerID="81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367" exitCode=0
Nov 01 20:21:50.128462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.128231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "kube-api-access-twj79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:21:50.128462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.128100    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" event={"ID":"bddffed5-308a-461f-bd60-1ce63aa6b4ba","Type":"ContainerDied","Data":"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"}
Nov 01 20:21:50.128462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.128312    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p" event={"ID":"bddffed5-308a-461f-bd60-1ce63aa6b4ba","Type":"ContainerDied","Data":"5aa858ba6e752df70c2b59bf237a45bd9ae8993ff7047d6813a25bf0fdce392b"}
Nov 01 20:21:50.128462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.128345    5041 scope.go:122] "RemoveContainer" containerID="81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"
Nov 01 20:21:50.129185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.128079    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"
Nov 01 20:21:50.130224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.130196    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy" pod "bddffed5-308a-461f-bd60-1ce63aa6b4ba" (UID: "bddffed5-308a-461f-bd60-1ce63aa6b4ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:21:50.179806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.157026    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.179806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.157067    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.179806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.157088    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.179806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.157103    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twj79\" (UniqueName: \"kubernetes.io/projected/bddffed5-308a-461f-bd60-1ce63aa6b4ba-kube-api-access-twj79\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:21:50.200030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.199913    5041 scope.go:122] "RemoveContainer" containerID="81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"
Nov 01 20:21:50.200577 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:50.200493    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367\": container with ID starting with 81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367 not found: ID does not exist" containerID="81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"
Nov 01 20:21:50.200577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.200531    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367"} err="failed to get container status \"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367\": rpc error: code = NotFound desc = could not find container \"81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367\": container with ID starting with 81b7625394ca5127f524e5c2429184e5d8cdadb54c70e68a27034a3c45244367 not found: ID does not exist"
Nov 01 20:21:50.486008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.485976    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:50.486267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.486238    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:21:50.491412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.491381    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-78j4p"]
Nov 01 20:21:50.498706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.498684    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:50.892013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.891979    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:50.892188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:50.892044    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:51.175879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:51.175765    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:21:51.924802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:51.899213    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:51.925219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:51.925178    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:52.431744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:52.431711    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" path="/var/lib/kubelet/pods/bddffed5-308a-461f-bd60-1ce63aa6b4ba/volumes"
Nov 01 20:21:52.892808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:52.892759    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:52.893352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:52.893314    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:53.892409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:53.892363    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:53.892872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:53.892431    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:54.892468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:54.892352    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 01 20:21:54.892468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:54.892410    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 01 20:21:55.899268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:55.899231    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:21:55.899826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:55.899609    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:21:55.900530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:55.900502    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:21:56.548888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:56.548854    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:21:56.549400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:56.549370    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy" containerID="cri-o://e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a" gracePeriod=30
Nov 01 20:21:57.174835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.174756    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:21:57.175567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175545    5041 cpu_manager.go:414] "containerMap: removing container" podUID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" containerName="payload-processing"
Nov 01 20:21:57.175567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175565    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" containerName="payload-processing"
Nov 01 20:21:57.175723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175617    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy"
Nov 01 20:21:57.175723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175623    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy"
Nov 01 20:21:57.175723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175631    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy"
Nov 01 20:21:57.175723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175635    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy"
Nov 01 20:21:57.176003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175746    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddffed5-308a-461f-bd60-1ce63aa6b4ba" containerName="istio-proxy"
Nov 01 20:21:57.176003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175755    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0867667c-ce38-4041-8ee5-a935a60d9401" containerName="istio-proxy"
Nov 01 20:21:57.176003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.175765    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f6c3fa-fa85-4df8-ac90-eac3ad554d39" containerName="payload-processing"
Nov 01 20:21:57.181054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.181026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.187846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.187820    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:21:57.190461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.190435    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"kube-root-ca.crt\""
Nov 01 20:21:57.190461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.190435    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"openshift-service-ca.crt\""
Nov 01 20:21:57.204795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.204216    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"default-dockercfg-w9jbz\""
Nov 01 20:21:57.204795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.204354    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"rate-test-model-036c69-kserve-self-signed-certs\""
Nov 01 20:21:57.298281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298243    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.298485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298442    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.298552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298493    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.298636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298618    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.298897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298877    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvq54\" (UniqueName: \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.298985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.298958    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.403871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.403817    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.404070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.403956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.404070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.404055    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.404225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.404090    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.404225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.404135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.404314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.404232    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvq54\" (UniqueName: \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.405059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.405023    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.405603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.405579    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.406074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.406008    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.407599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.407575    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.415704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.415670    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.416060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.416025    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvq54\" (UniqueName: \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\") pod \"rate-test-model-036c69-kserve-584595bf69-r99wt\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") " pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.507305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.507227    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:57.802112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.801429    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:21:57.822804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.820643    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:21:57.822804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.820794    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.833540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.829820    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-57d281-openshift-default-dockercfg-sjrpv\""
Nov 01 20:21:57.917427 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917395    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917541    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917618    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917725    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917839    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917901    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59ht\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.917986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.917942    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.918063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.918050    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:57.918149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:57.918137    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.020282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.020212    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.020659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.020637    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.021033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.020988    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.021228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.021210    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.021548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.021530    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59ht\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.021707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.021688    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.021895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.021870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.022109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.022093    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.022305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.022285    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.024798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.024086    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.025336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.025277    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.025587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.025548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.025768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.025603    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.025976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.025916    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.049221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.038958    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.049221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.042376    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.050913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.050834    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.058708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.058680    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59ht\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\") pod \"e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") " pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.113472 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:58.113441    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ee0f5ac_0817_4ee1_af87_afcebd2d9982.slice/crio-12047c3e83c08003193b5db38222387ade29261bf6210c92cd707f1d8b4abb0a WatchSource:0}: Error finding container 12047c3e83c08003193b5db38222387ade29261bf6210c92cd707f1d8b4abb0a: Status 404 returned error can't find the container with id 12047c3e83c08003193b5db38222387ade29261bf6210c92cd707f1d8b4abb0a
Nov 01 20:21:58.113731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.113706    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:21:58.160614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.160587    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:21:58.225403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.225317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" event={"ID":"3ee0f5ac-0817-4ee1-af87-afcebd2d9982","Type":"ContainerStarted","Data":"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"}
Nov 01 20:21:58.225403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.225369    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" event={"ID":"3ee0f5ac-0817-4ee1-af87-afcebd2d9982","Type":"ContainerStarted","Data":"12047c3e83c08003193b5db38222387ade29261bf6210c92cd707f1d8b4abb0a"}
Nov 01 20:21:58.249590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.249507    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" podStartSLOduration=1.24948807 podStartE2EDuration="1.24948807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:58.247343885 +0000 UTC m=+1405.968422563" watchObservedRunningTime="2026-11-01 20:21:58.24948807 +0000 UTC m=+1405.970566747"
Nov 01 20:21:58.453905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.453873    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:21:58.454068 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:21:58.454049    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1ed8cb_e203_49ca_a184_1ef8c85ce22b.slice/crio-b0a65884944697403d57da7e471852d9f67916aef1e70de7ec6f8d5e90136299 WatchSource:0}: Error finding container b0a65884944697403d57da7e471852d9f67916aef1e70de7ec6f8d5e90136299: Status 404 returned error can't find the container with id b0a65884944697403d57da7e471852d9f67916aef1e70de7ec6f8d5e90136299
Nov 01 20:21:58.456968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.456928    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:58.457079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.456999    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:58.457079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.457034    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:21:58.507590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:58.507566    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:21:58.548723 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:21:58.548690    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:21:59.236934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:59.236896    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerStarted","Data":"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"}
Nov 01 20:21:59.237435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:59.236954    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerStarted","Data":"b0a65884944697403d57da7e471852d9f67916aef1e70de7ec6f8d5e90136299"}
Nov 01 20:21:59.258145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:21:59.258098    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podStartSLOduration=2.258083771 podStartE2EDuration="2.258083771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:59.255225453 +0000 UTC m=+1406.976304150" watchObservedRunningTime="2026-11-01 20:21:59.258083771 +0000 UTC m=+1406.979162447"
Nov 01 20:22:00.161014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:00.160983    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:00.167104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:00.167074    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:00.245365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:00.245330    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:00.246378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:00.246359    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:02.017501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.017473    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:22:02.018020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.017998    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018069 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018042    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018112    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018156    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018241    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018309    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018362    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.018400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018374    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.018572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.018681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018658    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.018732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.018686    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:22:02.019261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.019236    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.019321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.019269    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.019321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.019286    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.019321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.019300    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.020454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.020432    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "kube-api-access-lqg2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:22:02.020632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.020450    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:22:02.020632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.020461    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:22:02.119975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.119940    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.120128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120066    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\") pod \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\" (UID: \"56a067d7-7f7b-4f13-b9af-f6358c85ba1b\") "
Nov 01 20:22:02.120364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120330    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.120520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120502    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.120565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120523    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.120565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120533    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.120565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.120544    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqg2s\" (UniqueName: \"kubernetes.io/projected/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-kube-api-access-lqg2s\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.122132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.122107    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy" pod "56a067d7-7f7b-4f13-b9af-f6358c85ba1b" (UID: "56a067d7-7f7b-4f13-b9af-f6358c85ba1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:22:02.222465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.222384    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a067d7-7f7b-4f13-b9af-f6358c85ba1b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:22:02.262591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.262559    5041 generic.go:356] "Generic (PLEG): container finished" podID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerID="e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a" exitCode=0
Nov 01 20:22:02.262770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.262606    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"
Nov 01 20:22:02.262770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.262638    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerDied","Data":"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"}
Nov 01 20:22:02.262770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.262699    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24" event={"ID":"56a067d7-7f7b-4f13-b9af-f6358c85ba1b","Type":"ContainerDied","Data":"6d7bce1a43c68f003c7e9f073773e281568fbab067df0ff62f36ab8ab41da787"}
Nov 01 20:22:02.262770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.262723    5041 scope.go:122] "RemoveContainer" containerID="e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"
Nov 01 20:22:02.277110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.277083    5041 scope.go:122] "RemoveContainer" containerID="313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"
Nov 01 20:22:02.296385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.296347    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:22:02.299058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.299036    5041 scope.go:122] "RemoveContainer" containerID="e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"
Nov 01 20:22:02.299729 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:22:02.299696    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a\": container with ID starting with e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a not found: ID does not exist" containerID="e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"
Nov 01 20:22:02.299875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.299732    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a"} err="failed to get container status \"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a\": rpc error: code = NotFound desc = could not find container \"e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a\": container with ID starting with e0b3971b9fb441e28cee1358c09f18f60e843e61e3ce94cf8bef9257d75b929a not found: ID does not exist"
Nov 01 20:22:02.299875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.299756    5041 scope.go:122] "RemoveContainer" containerID="313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"
Nov 01 20:22:02.300101 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:22:02.300084    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4\": container with ID starting with 313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4 not found: ID does not exist" containerID="313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"
Nov 01 20:22:02.300161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.300106    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4"} err="failed to get container status \"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4\": rpc error: code = NotFound desc = could not find container \"313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4\": container with ID starting with 313f6e41960e74c6bf76e745e676622dfc9ae8e70b076906481586b2bcdbe6f4 not found: ID does not exist"
Nov 01 20:22:02.304261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.304238    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-933245-openshift-default-8b8b4856c-nbw24"]
Nov 01 20:22:02.431241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:02.431201    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" path="/var/lib/kubelet/pods/56a067d7-7f7b-4f13-b9af-f6358c85ba1b/volumes"
Nov 01 20:22:04.796385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.796349    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:22:04.797079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.797061    5041 cpu_manager.go:414] "containerMap: removing container" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:04.797079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.797079    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:04.797233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.797101    5041 cpu_manager.go:414] "containerMap: removing container" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:04.797233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.797106    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:04.797335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.797258    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:04.802462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.802439    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.807022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.806994    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8b6cd6-openshift-default-dockercfg-tnb8t\""
Nov 01 20:22:04.809893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.809869    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:22:04.865011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.864977    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865031    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865104    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865222    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865247    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865283    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865301    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865327    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbbs\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.865361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.865351    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.974763    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.974851    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.974912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.974941    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.974972    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbbs\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.975008    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.975169    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.975213    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.975291    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.976244    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.976550    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.977232    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.977234    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.977993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.977515    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.979425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.979399    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.979951 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.979933    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.983353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.983330    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbbs\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:04.985256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:04.985236    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:05.037731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.037699    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:22:05.038592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.038571    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a067d7-7f7b-4f13-b9af-f6358c85ba1b" containerName="istio-proxy"
Nov 01 20:22:05.043582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.043565    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.048088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.048029    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a6873b-openshift-default-dockercfg-hfgdz\""
Nov 01 20:22:05.051678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.051650    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:22:05.076844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.076805    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077103    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077155    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077279    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kl7\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077353    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077392    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077467 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077421    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077542 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077521    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.077601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.077568    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.119608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.119565    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179154    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179206    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179247    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kl7\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179300    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179317    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179358    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179373    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.181953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.179433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.184289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.184243    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.184541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.184516    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.188806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.186942    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.191001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.190973    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.191108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.188662    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.206748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.206453    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.206748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.206698    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.208462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.208435    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kl7\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.208856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.208836    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.363854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.363738    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:05.407443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.407406    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:22:05.408302 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:22:05.408269    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe9fad5_626b_452a_b794_3c1c0cf872c3.slice/crio-101b4c6c0c20aec77d7dcf4616370025d45edf7e5ae1339462e3f50583382a71 WatchSource:0}: Error finding container 101b4c6c0c20aec77d7dcf4616370025d45edf7e5ae1339462e3f50583382a71: Status 404 returned error can't find the container with id 101b4c6c0c20aec77d7dcf4616370025d45edf7e5ae1339462e3f50583382a71
Nov 01 20:22:05.411962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.411918    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:05.412075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.412050    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:05.412171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.412102    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:05.656015 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:22:05.655987    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fa419f_07df_4189_ac37_a3b9217ae441.slice/crio-15a1c14c2f1b5fb215263e77c876235a520d83511510e310c2f591ec56dd105a WatchSource:0}: Error finding container 15a1c14c2f1b5fb215263e77c876235a520d83511510e310c2f591ec56dd105a: Status 404 returned error can't find the container with id 15a1c14c2f1b5fb215263e77c876235a520d83511510e310c2f591ec56dd105a
Nov 01 20:22:05.656274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.656252    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:22:05.659041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.659005    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:05.659145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.659080    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:05.659145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:05.659123    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:06.301270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.301235    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" event={"ID":"65fa419f-07df-4189-ac37-a3b9217ae441","Type":"ContainerStarted","Data":"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"}
Nov 01 20:22:06.301709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.301285    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" event={"ID":"65fa419f-07df-4189-ac37-a3b9217ae441","Type":"ContainerStarted","Data":"15a1c14c2f1b5fb215263e77c876235a520d83511510e310c2f591ec56dd105a"}
Nov 01 20:22:06.303175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.303149    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" event={"ID":"dfe9fad5-626b-452a-b794-3c1c0cf872c3","Type":"ContainerStarted","Data":"909ef718fa6d0c75b274107451f7b14fa04972779e26cf5df1325eb64f43a699"}
Nov 01 20:22:06.303297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.303188    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" event={"ID":"dfe9fad5-626b-452a-b794-3c1c0cf872c3","Type":"ContainerStarted","Data":"101b4c6c0c20aec77d7dcf4616370025d45edf7e5ae1339462e3f50583382a71"}
Nov 01 20:22:06.330813 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.329574    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podStartSLOduration=1.329551693 podStartE2EDuration="1.329551693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:06.325374765 +0000 UTC m=+1414.046453441" watchObservedRunningTime="2026-11-01 20:22:06.329551693 +0000 UTC m=+1414.050630371"
Nov 01 20:22:06.392798 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.392517    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:06.398943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.394240    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podStartSLOduration=2.394221228 podStartE2EDuration="2.394221228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:06.392183163 +0000 UTC m=+1414.113261840" watchObservedRunningTime="2026-11-01 20:22:06.394221228 +0000 UTC m=+1414.115299905"
Nov 01 20:22:06.398943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.396030    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:06.398943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:06.396072    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:07.120173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.120146    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:07.121657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.121632    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:22:07.121879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.121858    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 20:22:07.378862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.378258    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:07.378862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.378319    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:07.507591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.507556    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:22:07.520883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:07.520852    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:22:08.120876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:08.120844    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:22:08.121125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:08.121103    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 20:22:08.334035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:08.334007    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:22:08.370806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:08.364583    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:08.370806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:08.364658    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:09.119977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:09.119941    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:22:09.120405 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:09.119997    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 20:22:09.364801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:09.364211    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:09.364801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:09.364293    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:10.120561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:10.120524    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:22:10.121020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:10.120601    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 20:22:10.364271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:10.364230    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:10.364545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:10.364515    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:11.124300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:11.124260    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 01 20:22:11.124924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:11.124892    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 01 20:22:11.373580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:11.373546    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:11.373882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:11.373852    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:12.125696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:12.125662    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:12.126213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:12.126073    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:12.127151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:12.127122    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:22:12.364948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:12.364904    5041 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 01 20:22:12.365140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:12.364968    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 01 20:22:13.369418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:13.369386    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:13.369910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:13.369884    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:13.370845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:13.370820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:22:28.570379 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:22:28.570335    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:22:31.023239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:31.023200    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"]
Nov 01 20:22:31.027591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:31.027566    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e97b7dc-bs4t7"]
Nov 01 20:22:32.430625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:32.430594    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb687a5-8c6d-45ec-9840-15c582e4c58a" path="/var/lib/kubelet/pods/9fb687a5-8c6d-45ec-9840-15c582e4c58a/volumes"
Nov 01 20:22:35.371589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:35.371552    5041 scope.go:122] "RemoveContainer" containerID="5fc904696ddab961fdb5c6bc0a4c6e3e972e842f8679eabf25a73531c4c89495"
Nov 01 20:22:35.386149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:35.386124    5041 scope.go:122] "RemoveContainer" containerID="91a84058d339bdb746042a8199d4ac67983530a99b5a92764311b401b42a9ede"
Nov 01 20:22:35.404374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:35.404344    5041 scope.go:122] "RemoveContainer" containerID="237c8c0d55b87ed0f6fee808e2432ee211a47cfaa1a8a619a3d4cf1ecf25522b"
Nov 01 20:22:35.417696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:35.417663    5041 scope.go:122] "RemoveContainer" containerID="eb080219d2a6c2bc76b63de0295cb9672dfce825f9fd8cb53227dfc01f7f4ac9"
Nov 01 20:22:35.428481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:35.428455    5041 scope.go:122] "RemoveContainer" containerID="a51e44640973c70ea6442df87dc54add4ae9aaf860ec78430a7b9fb9f78f9ac7"
Nov 01 20:22:36.123758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.123695    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:22:36.131617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.131585    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.139067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.139031    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2babaef6-openshift-default-dockercfg-w2sjg\""
Nov 01 20:22:36.150295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.144137    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:22:36.193486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193447    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.193669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.193757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193667    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.193757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193682    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.193914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193748    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.193914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193843    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.194028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.194028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193972    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.194028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.193995    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.299827    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300002    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300044    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300080    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300187    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300361    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300411    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300446    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.300480    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.301678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.301578    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.302440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.301838    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.302440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.302070    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.302440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.302286    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.305477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.305409    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.339897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.328954    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.376931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.376846    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.377571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.377526    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.377753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.377734    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\") pod \"e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") " pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.453724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.453684    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:36.831120 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:22:36.831091    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90c22fc_b467_4b0c_a7b7_20b57ac9b6d7.slice/crio-56d0cdb2306362815d279793212ac3ab83432674ed39b87e8a3f99d03a60e2f5 WatchSource:0}: Error finding container 56d0cdb2306362815d279793212ac3ab83432674ed39b87e8a3f99d03a60e2f5: Status 404 returned error can't find the container with id 56d0cdb2306362815d279793212ac3ab83432674ed39b87e8a3f99d03a60e2f5
Nov 01 20:22:36.831311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.831285    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:22:36.834332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.834296    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:36.834416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.834378    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:36.834466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:36.834419    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:37.733648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:37.733610    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerStarted","Data":"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"}
Nov 01 20:22:37.734105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:37.733670    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerStarted","Data":"56d0cdb2306362815d279793212ac3ab83432674ed39b87e8a3f99d03a60e2f5"}
Nov 01 20:22:37.757824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:37.757762    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podStartSLOduration=1.757747642 podStartE2EDuration="1.757747642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:37.752048018 +0000 UTC m=+1445.473126694" watchObservedRunningTime="2026-11-01 20:22:37.757747642 +0000 UTC m=+1445.478826318"
Nov 01 20:22:38.454889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:38.454854    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:38.460352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:38.460327    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:38.759494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:38.759392    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:38.764875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:38.764850    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:22:41.069382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:41.069324    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"]
Nov 01 20:22:41.075433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:41.075394    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc34296f-mclzl"]
Nov 01 20:22:42.431278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:42.431245    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9336bd-7580-4b3e-9850-13ade5130ce1" path="/var/lib/kubelet/pods/fd9336bd-7580-4b3e-9850-13ade5130ce1/volumes"
Nov 01 20:22:50.946725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.946691    5041 generic.go:356] "Generic (PLEG): container finished" podID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerID="6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685" exitCode=0
Nov 01 20:22:50.947210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.946737    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerDied","Data":"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"}
Nov 01 20:22:50.947260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.947218    5041 scope.go:122] "RemoveContainer" containerID="6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"
Nov 01 20:22:50.948842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.948773    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:50.948912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.948893    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:50.949226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:50.948929    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:22:51.996038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:51.996002    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerStarted","Data":"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"}
Nov 01 20:22:52.162230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:52.162192    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:52.168840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:52.162530    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:52.169242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:52.169199    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:53.167058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:53.167011    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:53.167662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:53.167629    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:54.171014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:54.170974    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:54.171599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:54.171053    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:55.062804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:55.061843    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"]
Nov 01 20:22:55.091486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:55.091441    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a708d74a-7vpml"]
Nov 01 20:22:55.162260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:55.162163    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:55.162540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:55.162511    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:56.164814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:56.162218    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:56.164814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:56.162287    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:56.430306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:56.430215    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a8c445-40f9-48b6-9c74-090d20611d35" path="/var/lib/kubelet/pods/91a8c445-40f9-48b6-9c74-090d20611d35/volumes"
Nov 01 20:22:57.161802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:57.161674    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 01 20:22:57.161802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:57.161752    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 01 20:22:58.166856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:58.166823    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:58.167320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:58.167299    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:58.168048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:22:58.168031    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:22:58.543543 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:22:58.543504    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:23:03.875309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.875277    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"]
Nov 01 20:23:03.878931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.878912    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:03.884025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.883998    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:23:03.884539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.884112    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:23:03.884539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.884153    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:23:03.884539 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.884248    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:23:03.887543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:03.887514    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"]
Nov 01 20:23:04.043399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.043364    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.043596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.043408    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.043652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.043611    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbf7t\" (UniqueName: \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.144761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.144669    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbf7t\" (UniqueName: \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.144975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.144880    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.144975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.144918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.145764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.145735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.147870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.147843    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.153660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.153634    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbf7t\" (UniqueName: \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\") pod \"maas-api-revoke-sub-793758de-tbkmp\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.204563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.204534    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:04.532073 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:23:04.532035    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a66e20_aecc_435d_8d94_e07281b56717.slice/crio-9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf WatchSource:0}: Error finding container 9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf: Status 404 returned error can't find the container with id 9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf
Nov 01 20:23:04.532437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:04.532414    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"]
Nov 01 20:23:05.177982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:05.177939    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp" event={"ID":"62a66e20-aecc-435d-8d94-e07281b56717","Type":"ContainerStarted","Data":"f5ef33196764fcfe3601f3690a6c89fd95106bed9b86a12349afdf98b35db892"}
Nov 01 20:23:05.178389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:05.178030    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp" event={"ID":"62a66e20-aecc-435d-8d94-e07281b56717","Type":"ContainerStarted","Data":"9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf"}
Nov 01 20:23:05.196142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:05.196090    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp" podStartSLOduration=2.196077699 podStartE2EDuration="2.196077699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:05.194893594 +0000 UTC m=+1472.915972269" watchObservedRunningTime="2026-11-01 20:23:05.196077699 +0000 UTC m=+1472.917156374"
Nov 01 20:23:06.023582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:06.023507    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"]
Nov 01 20:23:06.027939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:06.027913    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a741fe3e-g2rkx"]
Nov 01 20:23:06.189889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:06.189854    5041 generic.go:356] "Generic (PLEG): container finished" podID="62a66e20-aecc-435d-8d94-e07281b56717" containerID="f5ef33196764fcfe3601f3690a6c89fd95106bed9b86a12349afdf98b35db892" exitCode=0
Nov 01 20:23:06.190321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:06.189902    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp" event={"ID":"62a66e20-aecc-435d-8d94-e07281b56717","Type":"ContainerDied","Data":"f5ef33196764fcfe3601f3690a6c89fd95106bed9b86a12349afdf98b35db892"}
Nov 01 20:23:06.430512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:06.430479    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0c57a6-4c28-495e-b871-86ba2622683a" path="/var/lib/kubelet/pods/3f0c57a6-4c28-495e-b871-86ba2622683a/volumes"
Nov 01 20:23:07.387456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.387425    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:07.410595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.410566    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\") pod \"62a66e20-aecc-435d-8d94-e07281b56717\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") "
Nov 01 20:23:07.410855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.410835    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\") pod \"62a66e20-aecc-435d-8d94-e07281b56717\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") "
Nov 01 20:23:07.410950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.410877    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\" (UniqueName: \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\") pod \"62a66e20-aecc-435d-8d94-e07281b56717\" (UID: \"62a66e20-aecc-435d-8d94-e07281b56717\") "
Nov 01 20:23:07.411236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.411208    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca" pod "62a66e20-aecc-435d-8d94-e07281b56717" (UID: "62a66e20-aecc-435d-8d94-e07281b56717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:07.411613 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.411587    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.412879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.412849    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca" pod "62a66e20-aecc-435d-8d94-e07281b56717" (UID: "62a66e20-aecc-435d-8d94-e07281b56717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:07.413245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.413221    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t" pod "62a66e20-aecc-435d-8d94-e07281b56717" (UID: "62a66e20-aecc-435d-8d94-e07281b56717"). InnerVolumeSpecName "kube-api-access-cbf7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:07.513334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.513244    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbf7t\" (UniqueName: \"kubernetes.io/projected/62a66e20-aecc-435d-8d94-e07281b56717-kube-api-access-cbf7t\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:07.513334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:07.513276    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a66e20-aecc-435d-8d94-e07281b56717-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:08.209616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:08.209585    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"
Nov 01 20:23:08.209828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:08.209612    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp" event={"ID":"62a66e20-aecc-435d-8d94-e07281b56717","Type":"ContainerDied","Data":"9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf"}
Nov 01 20:23:08.209828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:08.209665    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9681bf245c1b26fa6807affc8f184c6b13ff5c54fc0d2ec4a6a359d80bbd9caf"
Nov 01 20:23:18.070871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:18.068830    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"]
Nov 01 20:23:18.094802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:18.093652    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059ea56a-4vzcv"]
Nov 01 20:23:18.430218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:18.430085    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7" path="/var/lib/kubelet/pods/f5bf85b3-9d21-4a63-ae3c-adc8c14f3ee7/volumes"
Nov 01 20:23:24.801944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:24.801887    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:23:24.802388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:24.801962    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:23:25.586008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.585974    5041 generic.go:356] "Generic (PLEG): container finished" podID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerID="2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82" exitCode=0
Nov 01 20:23:25.586219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.586041    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerDied","Data":"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"}
Nov 01 20:23:25.586474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.586460    5041 scope.go:122] "RemoveContainer" containerID="2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"
Nov 01 20:23:25.588306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.588275    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:23:25.588416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.588358    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:23:25.588480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:25.588410    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:23:26.597039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:26.597001    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerStarted","Data":"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"}
Nov 01 20:23:27.464803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:27.464161    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:23:27.464803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:27.464372    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 01 20:23:27.464803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:27.464430    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 01 20:23:28.059892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:28.059856    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"]
Nov 01 20:23:28.083846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:28.083816    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9ad4fad-n69n4"]
Nov 01 20:23:28.431443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:28.431358    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8" path="/var/lib/kubelet/pods/a1ccfed8-4f8f-412b-9ef8-69ce0efe71f8/volumes"
Nov 01 20:23:28.466998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:28.466595    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 01 20:23:28.466998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:28.466668    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 01 20:23:28.563537 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:23:28.563456    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:23:29.470006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:29.469969    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 01 20:23:29.470545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:29.470052    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 01 20:23:30.564490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:30.564441    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 01 20:23:30.574413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:30.574356    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 01 20:23:31.461801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:31.461166    5041 patch_prober.go:29] interesting pod/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 01 20:23:31.461801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:31.461247    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 01 20:23:32.804425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:32.804395    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:23:32.805010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:32.804991    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:23:32.994468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:32.994435    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:23:32.995403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:32.995369    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:23:32.996037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:32.996014    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:23:35.530303 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.530272    5041 scope.go:122] "RemoveContainer" containerID="3689da22b997de044ba6f75ed65c517f0b130665f636e9f215c8b00ce13890bd"
Nov 01 20:23:35.573729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.573686    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"]
Nov 01 20:23:35.574369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.574351    5041 cpu_manager.go:414] "containerMap: removing container" podUID="62a66e20-aecc-435d-8d94-e07281b56717" containerName="revoke-keys"
Nov 01 20:23:35.574369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.574369    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a66e20-aecc-435d-8d94-e07281b56717" containerName="revoke-keys"
Nov 01 20:23:35.574520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.574511    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a66e20-aecc-435d-8d94-e07281b56717" containerName="revoke-keys"
Nov 01 20:23:35.578095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.578070    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.581751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.581733    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:23:35.582000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.581750    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:23:35.583186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.583161    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:23:35.583361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.583185    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:23:35.584894 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.584868    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"]
Nov 01 20:23:35.652791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.652741    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.653016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.652993    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.653147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.653124    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26cs\" (UniqueName: \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.754108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.754074    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.754277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.754196    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.754277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.754243    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26cs\" (UniqueName: \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.754909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.754878    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.756699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.756675    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.761898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.761874    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26cs\" (UniqueName: \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\") pod \"maas-api-revoke-sub-dd94e543-qh6tt\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:35.896323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:35.896293    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:36.177319 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:23:36.177289    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d14408_e902_4289_a2b7_875aa970c8e4.slice/crio-fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31 WatchSource:0}: Error finding container fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31: Status 404 returned error can't find the container with id fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31
Nov 01 20:23:36.178146 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:36.178115    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"]
Nov 01 20:23:36.770534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:36.770429    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt" event={"ID":"12d14408-e902-4289-a2b7-875aa970c8e4","Type":"ContainerStarted","Data":"a7524d1935f49f2ae76f9ae2cbbe4369335cdcc7b378a28cc1d00daf396c5a52"}
Nov 01 20:23:36.770534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:36.770504    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt" event={"ID":"12d14408-e902-4289-a2b7-875aa970c8e4","Type":"ContainerStarted","Data":"fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31"}
Nov 01 20:23:36.791412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:36.791356    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt" podStartSLOduration=1.79134175 podStartE2EDuration="1.79134175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:36.789910815 +0000 UTC m=+1504.510989491" watchObservedRunningTime="2026-11-01 20:23:36.79134175 +0000 UTC m=+1504.512420427"
Nov 01 20:23:37.780736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:37.780702    5041 generic.go:356] "Generic (PLEG): container finished" podID="12d14408-e902-4289-a2b7-875aa970c8e4" containerID="a7524d1935f49f2ae76f9ae2cbbe4369335cdcc7b378a28cc1d00daf396c5a52" exitCode=0
Nov 01 20:23:37.781170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:37.780802    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt" event={"ID":"12d14408-e902-4289-a2b7-875aa970c8e4","Type":"ContainerDied","Data":"a7524d1935f49f2ae76f9ae2cbbe4369335cdcc7b378a28cc1d00daf396c5a52"}
Nov 01 20:23:39.012620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.012593    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:39.148134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.148042    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\" (UniqueName: \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\") pod \"12d14408-e902-4289-a2b7-875aa970c8e4\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") "
Nov 01 20:23:39.148296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.148191    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\") pod \"12d14408-e902-4289-a2b7-875aa970c8e4\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") "
Nov 01 20:23:39.148381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.148355    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\") pod \"12d14408-e902-4289-a2b7-875aa970c8e4\" (UID: \"12d14408-e902-4289-a2b7-875aa970c8e4\") "
Nov 01 20:23:39.148873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.148840    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca" pod "12d14408-e902-4289-a2b7-875aa970c8e4" (UID: "12d14408-e902-4289-a2b7-875aa970c8e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:39.149099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.149072    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:39.150924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.150892    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs" pod "12d14408-e902-4289-a2b7-875aa970c8e4" (UID: "12d14408-e902-4289-a2b7-875aa970c8e4"). InnerVolumeSpecName "kube-api-access-p26cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:39.150924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.150904    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca" pod "12d14408-e902-4289-a2b7-875aa970c8e4" (UID: "12d14408-e902-4289-a2b7-875aa970c8e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:39.250984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.250946    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d14408-e902-4289-a2b7-875aa970c8e4-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:39.250984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.250983    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p26cs\" (UniqueName: \"kubernetes.io/projected/12d14408-e902-4289-a2b7-875aa970c8e4-kube-api-access-p26cs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:39.588995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.588957    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:23:39.589248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.589220    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" containerID="cri-o://392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d" gracePeriod=30
Nov 01 20:23:39.857927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.834139    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt" event={"ID":"12d14408-e902-4289-a2b7-875aa970c8e4","Type":"ContainerDied","Data":"fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31"}
Nov 01 20:23:39.857927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.834202    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6333b799806b7513f13fdf16ea5ff3bee018f134b548eede0f875d4f746f31"
Nov 01 20:23:39.857927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:39.834269    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"
Nov 01 20:23:43.170088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:43.170050    5041 patch_prober.go:29] interesting pod/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:23:43.170564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:43.170123    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:23:44.899996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.899969    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:23:44.910142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910117    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910162    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910252    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910299    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910414    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910511    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910539    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:44.910586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910569    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910597    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910607    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:44.910681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910520    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:44.910681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910634    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\") pod \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\" (UID: \"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b\") "
Nov 01 20:23:44.910954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.910927    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:44.911076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911052    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:44.911531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911503    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:44.911643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911535    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:44.911643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911553    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:44.911643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911567    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:44.911643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.911581    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:44.912671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.912650    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:23:44.912843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.912826    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "kube-api-access-g59ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:44.912913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.912843    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:23:44.912955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:44.912915    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token" pod "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" (UID: "3c1ed8cb-e203-49ca-a184-1ef8c85ce22b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:45.013651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.013617    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:45.013651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.013648    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:45.013651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.013658    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g59ht\" (UniqueName: \"kubernetes.io/projected/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-kube-api-access-g59ht\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:45.013903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.013668    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:45.065097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.065057    5041 generic.go:356] "Generic (PLEG): container finished" podID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerID="392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d" exitCode=0
Nov 01 20:23:45.065291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.065120    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"
Nov 01 20:23:45.065291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.065159    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerDied","Data":"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"}
Nov 01 20:23:45.065291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.065227    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b" event={"ID":"3c1ed8cb-e203-49ca-a184-1ef8c85ce22b","Type":"ContainerDied","Data":"b0a65884944697403d57da7e471852d9f67916aef1e70de7ec6f8d5e90136299"}
Nov 01 20:23:45.065291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.065259    5041 scope.go:122] "RemoveContainer" containerID="392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"
Nov 01 20:23:45.079501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.079479    5041 scope.go:122] "RemoveContainer" containerID="6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"
Nov 01 20:23:45.096522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.096494    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:23:45.097599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.097581    5041 scope.go:122] "RemoveContainer" containerID="392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"
Nov 01 20:23:45.097947 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:23:45.097925    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d\": container with ID starting with 392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d not found: ID does not exist" containerID="392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"
Nov 01 20:23:45.098025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.097955    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d"} err="failed to get container status \"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d\": rpc error: code = NotFound desc = could not find container \"392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d\": container with ID starting with 392e79061bb446f88f178b31dec7107d1f6ff5575ef459101fb308ef4bb2ea5d not found: ID does not exist"
Nov 01 20:23:45.098025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.097972    5041 scope.go:122] "RemoveContainer" containerID="6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"
Nov 01 20:23:45.098272 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:23:45.098245    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685\": container with ID starting with 6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685 not found: ID does not exist" containerID="6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"
Nov 01 20:23:45.098321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.098283    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685"} err="failed to get container status \"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685\": rpc error: code = NotFound desc = could not find container \"6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685\": container with ID starting with 6cd9003965c5704be0fc061a18eaf081107e2df5077d12eebd96419555d0d685 not found: ID does not exist"
Nov 01 20:23:45.102430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.102409    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-57d281-openshift-default-697dd4cd45-xdq2b"]
Nov 01 20:23:45.759904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.759867    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"]
Nov 01 20:23:45.760522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760509    5041 cpu_manager.go:414] "containerMap: removing container" podUID="12d14408-e902-4289-a2b7-875aa970c8e4" containerName="revoke-keys"
Nov 01 20:23:45.760580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760523    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d14408-e902-4289-a2b7-875aa970c8e4" containerName="revoke-keys"
Nov 01 20:23:45.760580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760548    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:45.760580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760553    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:45.760694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760676    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:45.760694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.760689    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d14408-e902-4289-a2b7-875aa970c8e4" containerName="revoke-keys"
Nov 01 20:23:45.764195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.764174    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.768861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.768838    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:23:45.768861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.768846    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:23:45.769059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.768948    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:23:45.769165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.769137    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:23:45.771619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.771594    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"]
Nov 01 20:23:45.830930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.830891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x2wf\" (UniqueName: \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.831131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.831059    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.831131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.831110    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.933330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.933297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x2wf\" (UniqueName: \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.933763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.933400    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.933763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.933432    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.934222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.934198    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.935999 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.935977    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:45.944701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:45.944673    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x2wf\" (UniqueName: \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\") pod \"maas-api-revoke-sub-b31bb19b-tvfgw\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:46.081669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:46.081637    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:46.363623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:46.363595    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"]
Nov 01 20:23:46.365210 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:23:46.365181    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ea781a_13ed_4ee9_99bd_d810a4f6efde.slice/crio-d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652 WatchSource:0}: Error finding container d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652: Status 404 returned error can't find the container with id d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652
Nov 01 20:23:46.430232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:46.430205    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" path="/var/lib/kubelet/pods/3c1ed8cb-e203-49ca-a184-1ef8c85ce22b/volumes"
Nov 01 20:23:47.085996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:47.085956    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw" event={"ID":"76ea781a-13ed-4ee9-99bd-d810a4f6efde","Type":"ContainerStarted","Data":"f2ddbafd6bbbf8e5267c7c7a42ec49fc719bb9c3c80dea80c6e0dfcd13154e09"}
Nov 01 20:23:47.086407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:47.086017    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw" event={"ID":"76ea781a-13ed-4ee9-99bd-d810a4f6efde","Type":"ContainerStarted","Data":"d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652"}
Nov 01 20:23:47.105407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:47.105359    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw" podStartSLOduration=2.105346334 podStartE2EDuration="2.105346334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:47.100211961 +0000 UTC m=+1514.821290632" watchObservedRunningTime="2026-11-01 20:23:47.105346334 +0000 UTC m=+1514.826425010"
Nov 01 20:23:48.095029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:48.094990    5041 generic.go:356] "Generic (PLEG): container finished" podID="76ea781a-13ed-4ee9-99bd-d810a4f6efde" containerID="f2ddbafd6bbbf8e5267c7c7a42ec49fc719bb9c3c80dea80c6e0dfcd13154e09" exitCode=0
Nov 01 20:23:48.095933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:48.095079    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw" event={"ID":"76ea781a-13ed-4ee9-99bd-d810a4f6efde","Type":"ContainerDied","Data":"f2ddbafd6bbbf8e5267c7c7a42ec49fc719bb9c3c80dea80c6e0dfcd13154e09"}
Nov 01 20:23:49.279329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.279301    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:49.393597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.393511    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\") pod \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") "
Nov 01 20:23:49.393731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.393600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\") pod \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") "
Nov 01 20:23:49.393792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.393729    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\" (UniqueName: \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\") pod \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\" (UID: \"76ea781a-13ed-4ee9-99bd-d810a4f6efde\") "
Nov 01 20:23:49.393909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.393885    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca" pod "76ea781a-13ed-4ee9-99bd-d810a4f6efde" (UID: "76ea781a-13ed-4ee9-99bd-d810a4f6efde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:23:49.394246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.394223    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:49.395704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.395677    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca" pod "76ea781a-13ed-4ee9-99bd-d810a4f6efde" (UID: "76ea781a-13ed-4ee9-99bd-d810a4f6efde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:23:49.395825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.395747    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf" pod "76ea781a-13ed-4ee9-99bd-d810a4f6efde" (UID: "76ea781a-13ed-4ee9-99bd-d810a4f6efde"). InnerVolumeSpecName "kube-api-access-2x2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:23:49.495731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.495695    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ea781a-13ed-4ee9-99bd-d810a4f6efde-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:49.495731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:49.495727    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x2wf\" (UniqueName: \"kubernetes.io/projected/76ea781a-13ed-4ee9-99bd-d810a4f6efde-kube-api-access-2x2wf\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:23:50.112647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:50.112619    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"
Nov 01 20:23:50.112842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:50.112615    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw" event={"ID":"76ea781a-13ed-4ee9-99bd-d810a4f6efde","Type":"ContainerDied","Data":"d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652"}
Nov 01 20:23:50.112842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:50.112738    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0db66a91e0d0d7918e9ea7b5018942970fce4e152106d47e2b222e458c3b652"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.661369    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662426    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662446    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662474    5041 cpu_manager.go:414] "containerMap: removing container" podUID="76ea781a-13ed-4ee9-99bd-d810a4f6efde" containerName="revoke-keys"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662482    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ea781a-13ed-4ee9-99bd-d810a4f6efde" containerName="revoke-keys"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662698    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ea781a-13ed-4ee9-99bd-d810a4f6efde" containerName="revoke-keys"
Nov 01 20:23:54.675803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.662712    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1ed8cb-e203-49ca-a184-1ef8c85ce22b" containerName="istio-proxy"
Nov 01 20:23:54.683754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.683133    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.683960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.683494    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:23:54.690042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.688749    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-82e760\"/\"default-dockercfg-9mzht\""
Nov 01 20:23:54.690042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.689086    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-82e760\"/\"openshift-service-ca.crt\""
Nov 01 20:23:54.690042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.689161    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-82e760\"/\"noten-model-82e760-kserve-self-signed-certs\""
Nov 01 20:23:54.690042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.689744    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-82e760\"/\"kube-root-ca.crt\""
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.834770    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.834922    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnnc\" (UniqueName: \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.834965    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.835025    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.835184    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.840802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.835229    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937047    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937122    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937238    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937356    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnnc\" (UniqueName: \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937409    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937463    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.937703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.937688    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.938303 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.938276    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.938547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.938528    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.940657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.940605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.941607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.941582    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:54.952476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:54.952374    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnnc\" (UniqueName: \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\") pod \"noten-model-82e760-kserve-5689c9d8f7-js9hr\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") " pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:55.020471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:55.020432    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:55.638185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:55.638115    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:23:56.287959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:56.280886    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" event={"ID":"e6890fec-3513-4287-9d3a-7f05b9443408","Type":"ContainerStarted","Data":"86742bf06f09fc6c6ba14be825ac966bb51eb95dd23375ce15cf5a0a5114b85c"}
Nov 01 20:23:56.287959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:56.280937    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" event={"ID":"e6890fec-3513-4287-9d3a-7f05b9443408","Type":"ContainerStarted","Data":"24492b30cd14164ed4784325c52884a1170db42b210c97cb9b840570f701d894"}
Nov 01 20:23:56.312799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:56.298402    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" podStartSLOduration=2.298384531 podStartE2EDuration="2.298384531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:56.297207198 +0000 UTC m=+1524.018285875" watchObservedRunningTime="2026-11-01 20:23:56.298384531 +0000 UTC m=+1524.019463209"
Nov 01 20:23:57.020966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:23:57.020935    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:23:58.544840 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:23:58.544799    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:24:00.292802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:00.292309    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:24:12.721164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.721127    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:24:12.725648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.725623    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.736257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.736218    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:24:12.881796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881597    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.881796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.881796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881702    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjjd\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881823    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881856    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881890    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.881975    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.882016    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:12.882124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:12.882079    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.023242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023094    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.023242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023178    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.023242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023209    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjjd\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023354    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023402    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023438    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023560    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023617    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.024085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.023694    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.026458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.025141    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.026458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.026111    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.026458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.026138    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.026458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.026411    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.026729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.026632    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.030171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.030122    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.036311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.035407    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.036613 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.036590    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.037567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.037538    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjjd\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.057165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.054159    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:13.379595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.379566    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:24:13.379723 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:13.379559    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44650ec9_6c47_4dea_9f35_d807985ab580.slice/crio-e9a58a26acb1a069195e3b5824f527d11b4c1ba8c256fd8d58037acf107f3158 WatchSource:0}: Error finding container e9a58a26acb1a069195e3b5824f527d11b4c1ba8c256fd8d58037acf107f3158: Status 404 returned error can't find the container with id e9a58a26acb1a069195e3b5824f527d11b4c1ba8c256fd8d58037acf107f3158
Nov 01 20:24:13.382749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.382716    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:13.382897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.382829    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:13.382941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.382896    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:13.509062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.509023    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" event={"ID":"44650ec9-6c47-4dea-9f35-d807985ab580","Type":"ContainerStarted","Data":"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"}
Nov 01 20:24:13.509237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.509079    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" event={"ID":"44650ec9-6c47-4dea-9f35-d807985ab580","Type":"ContainerStarted","Data":"e9a58a26acb1a069195e3b5824f527d11b4c1ba8c256fd8d58037acf107f3158"}
Nov 01 20:24:13.536228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:13.536173    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podStartSLOduration=1.5361590330000001 podStartE2EDuration="1.536159033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:13.528635014 +0000 UTC m=+1541.249713719" watchObservedRunningTime="2026-11-01 20:24:13.536159033 +0000 UTC m=+1541.257237709"
Nov 01 20:24:14.055200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:14.055160    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:14.057202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:14.057174    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:14.057359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:14.057233    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:15.066826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.062226    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:15.066826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.062291    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:15.320372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.320281    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" podUID="e6890fec-3513-4287-9d3a-7f05b9443408" containerName="main" containerID="cri-o://86742bf06f09fc6c6ba14be825ac966bb51eb95dd23375ce15cf5a0a5114b85c" gracePeriod=45
Nov 01 20:24:15.540854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.534882    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6890fec-3513-4287-9d3a-7f05b9443408" containerID="86742bf06f09fc6c6ba14be825ac966bb51eb95dd23375ce15cf5a0a5114b85c" exitCode=0
Nov 01 20:24:15.540854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.535242    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" event={"ID":"e6890fec-3513-4287-9d3a-7f05b9443408","Type":"ContainerDied","Data":"86742bf06f09fc6c6ba14be825ac966bb51eb95dd23375ce15cf5a0a5114b85c"}
Nov 01 20:24:15.735698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.735673    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:24:15.807238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.807210    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.807543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.807531    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\" (UniqueName: \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.807688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.807672    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.807882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.807865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.808027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.808011    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.808157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.808142    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\") pod \"e6890fec-3513-4287-9d3a-7f05b9443408\" (UID: \"e6890fec-3513-4287-9d3a-7f05b9443408\") "
Nov 01 20:24:15.808566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.808531    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.810714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.810582    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.810948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.810898    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.811276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.811225    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.833648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.833611    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "kube-api-access-kfnnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:15.833919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.833895    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:15.836047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.836017    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs" pod "e6890fec-3513-4287-9d3a-7f05b9443408" (UID: "e6890fec-3513-4287-9d3a-7f05b9443408"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:15.912721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.912690    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.912721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.912721    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnnc\" (UniqueName: \"kubernetes.io/projected/e6890fec-3513-4287-9d3a-7f05b9443408-kube-api-access-kfnnc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.912946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.912732    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.912946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.912743    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6890fec-3513-4287-9d3a-7f05b9443408-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:15.912946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:15.912752    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6890fec-3513-4287-9d3a-7f05b9443408-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:16.067805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.060215    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:16.067805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.060298    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:16.393472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.393384    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:24:16.394017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.393968    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy" containerID="cri-o://1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0" gracePeriod=30
Nov 01 20:24:16.578465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.578427    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr" event={"ID":"e6890fec-3513-4287-9d3a-7f05b9443408","Type":"ContainerDied","Data":"24492b30cd14164ed4784325c52884a1170db42b210c97cb9b840570f701d894"}
Nov 01 20:24:16.578744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.578723    5041 scope.go:122] "RemoveContainer" containerID="86742bf06f09fc6c6ba14be825ac966bb51eb95dd23375ce15cf5a0a5114b85c"
Nov 01 20:24:16.579132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.579113    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"
Nov 01 20:24:16.647899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.647805    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:24:16.761802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:16.761268    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-82e760/noten-model-82e760-kserve-5689c9d8f7-js9hr"]
Nov 01 20:24:17.067930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:17.067885    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:17.068381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:17.067959    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:18.066818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:18.066306    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:18.066818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:18.066373    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:18.522859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:18.508321    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6890fec-3513-4287-9d3a-7f05b9443408" path="/var/lib/kubelet/pods/e6890fec-3513-4287-9d3a-7f05b9443408/volumes"
Nov 01 20:24:19.066876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:19.066188    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:19.066876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:19.066246    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:20.054994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:20.054952    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:20.055468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:20.055014    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:21.059410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.059370    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 01 20:24:21.060002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.059952    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 01 20:24:21.725942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.725907    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:24:21.777467 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777426    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777562    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777624    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777743    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777822    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777907    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.777983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.777974    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.778234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.778037    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\") pod \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\" (UID: \"5f5407d2-caa7-4795-a92a-df01bd5a8e9b\") "
Nov 01 20:24:21.791866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.791823    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.795290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.795106    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.797868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.795482    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.797868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.796155    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:24:21.800358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.798187    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.800358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.799378    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:21.800358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.799678    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:21.805989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.805957    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:21.806126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.806049    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7" pod "5f5407d2-caa7-4795-a92a-df01bd5a8e9b" (UID: "5f5407d2-caa7-4795-a92a-df01bd5a8e9b"). InnerVolumeSpecName "kube-api-access-bznx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:21.824402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.824359    5041 generic.go:356] "Generic (PLEG): container finished" podID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerID="1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0" exitCode=0
Nov 01 20:24:21.824556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.824436    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" event={"ID":"5f5407d2-caa7-4795-a92a-df01bd5a8e9b","Type":"ContainerDied","Data":"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"}
Nov 01 20:24:21.824627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.824606    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78" event={"ID":"5f5407d2-caa7-4795-a92a-df01bd5a8e9b","Type":"ContainerDied","Data":"44281c1d442d0b2a77c0a8dedc9f99103fdf67ffeb16282faf55728a52e00378"}
Nov 01 20:24:21.824696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.824468    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"
Nov 01 20:24:21.824880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.824645    5041 scope.go:122] "RemoveContainer" containerID="1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"
Nov 01 20:24:21.857507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.857203    5041 scope.go:122] "RemoveContainer" containerID="1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"
Nov 01 20:24:21.857831 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:21.857716    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0\": container with ID starting with 1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0 not found: ID does not exist" containerID="1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"
Nov 01 20:24:21.857831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.857753    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0"} err="failed to get container status \"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0\": rpc error: code = NotFound desc = could not find container \"1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0\": container with ID starting with 1127b46f61d81659e6fd9cd7a6970e80d00b239cdb5931aca8cca0cc8e3503b0 not found: ID does not exist"
Nov 01 20:24:21.870770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.870741    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890194    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890228    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bznx7\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-kube-api-access-bznx7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890248    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890262    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890275    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890292    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890305    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890320    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.890357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.890333    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5407d2-caa7-4795-a92a-df01bd5a8e9b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:21.900171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:21.900142    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaea41-openshift-default-67cb954d5-t2j78"]
Nov 01 20:24:22.464376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:22.464335    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" path="/var/lib/kubelet/pods/5f5407d2-caa7-4795-a92a-df01bd5a8e9b/volumes"
Nov 01 20:24:23.086051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:23.086012    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:24:23.086270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:23.086087    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:24:23.119812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:23.119227    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:23.124800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:23.121446    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:23.142830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:23.140091    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:24:27.751141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.751104    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:24:27.752081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752056    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6890fec-3513-4287-9d3a-7f05b9443408" containerName="main"
Nov 01 20:24:27.752081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752081    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6890fec-3513-4287-9d3a-7f05b9443408" containerName="main"
Nov 01 20:24:27.752249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752153    5041 cpu_manager.go:414] "containerMap: removing container" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy"
Nov 01 20:24:27.752249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752161    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy"
Nov 01 20:24:27.752337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752323    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5407d2-caa7-4795-a92a-df01bd5a8e9b" containerName="istio-proxy"
Nov 01 20:24:27.752375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.752348    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6890fec-3513-4287-9d3a-7f05b9443408" containerName="main"
Nov 01 20:24:27.758194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.758172    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.781839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.780118    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:24:27.914309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914276    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914316    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c888f\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914352    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914548    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914613    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914645    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914677    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914754    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:27.914911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:27.914883    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016297    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c888f\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016363    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016414    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016451    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016517    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016557    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.016691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016685    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.017047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016914    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.017047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.016985    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.017047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.017034    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.017190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.017092    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.017499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.017475    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.019142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.019117    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.019240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.019217    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.024002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.023975    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.024092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.024052    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c888f\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.086598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.086573    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:28.376917 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:28.376880    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05fc0dc_efeb_4b19_a66a_9e3431c87052.slice/crio-2f0ecde50593bbf4712e4fc5c105cdefa954a0357c11bed92fe8e2b7aed7dd8b WatchSource:0}: Error finding container 2f0ecde50593bbf4712e4fc5c105cdefa954a0357c11bed92fe8e2b7aed7dd8b: Status 404 returned error can't find the container with id 2f0ecde50593bbf4712e4fc5c105cdefa954a0357c11bed92fe8e2b7aed7dd8b
Nov 01 20:24:28.380056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.380012    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:28.380143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.380106    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:28.380186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.380147    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:24:28.380284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.380258    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:24:28.546969 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:28.546928    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:24:28.898210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.898170    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" event={"ID":"a05fc0dc-efeb-4b19-a66a-9e3431c87052","Type":"ContainerStarted","Data":"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"}
Nov 01 20:24:28.898829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.898764    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" event={"ID":"a05fc0dc-efeb-4b19-a66a-9e3431c87052","Type":"ContainerStarted","Data":"2f0ecde50593bbf4712e4fc5c105cdefa954a0357c11bed92fe8e2b7aed7dd8b"}
Nov 01 20:24:28.921286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:28.921231    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podStartSLOduration=1.9212160379999998 podStartE2EDuration="1.921216038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:28.916670964 +0000 UTC m=+1556.637749651" watchObservedRunningTime="2026-11-01 20:24:28.921216038 +0000 UTC m=+1556.642294713"
Nov 01 20:24:29.087600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:29.087563    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:29.089597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:29.089571    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:29.089741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:29.089624    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:30.086975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:30.086941    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:30.087372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:30.086999    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:31.087265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:31.087232    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:31.087672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:31.087285    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:32.087419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:32.087379    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:32.088012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:32.087474    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:33.087484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:33.087448    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:33.087956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:33.087524    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:34.087971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:34.087938    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:34.088389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:34.088001    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:35.088795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.088747    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 01 20:24:35.089265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.088824    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 01 20:24:35.429276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.429183    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"]
Nov 01 20:24:35.435723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.435693    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.441495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.441469    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:24:35.441703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.441490    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:24:35.441703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.441530    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:24:35.442400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.441589    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:24:35.443186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.443160    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"]
Nov 01 20:24:35.515513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.515469    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.515714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.515533    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgq4s\" (UniqueName: \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.515714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.515689    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.562914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.562870    5041 scope.go:122] "RemoveContainer" containerID="56deef347f66390b256bf195d578cb5418446716ff01240da37c5e40b8f6cacf"
Nov 01 20:24:35.580352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.580188    5041 scope.go:122] "RemoveContainer" containerID="3dfb5991b6e104f2ef6bbe9e0ef88f773abc7dc0f61556756f7bcbadcf7c5566"
Nov 01 20:24:35.591677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.591649    5041 scope.go:122] "RemoveContainer" containerID="5a953db70d0bac243be1040f99a8a241157cf2c0792330eb3509ff131d6ecbb8"
Nov 01 20:24:35.604509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.604490    5041 scope.go:122] "RemoveContainer" containerID="fd09206139d2813e8a3c67a2a279fca993a86af78196aeb2c2b366b040578bc2"
Nov 01 20:24:35.617434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.617403    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.617585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.617473    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgq4s\" (UniqueName: \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.617585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.617574    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.618662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.618644    5041 scope.go:122] "RemoveContainer" containerID="4a37290c78a0aff0b7f33b01b8fc3e8e5d4c8e25541e915f79195b569b2461eb"
Nov 01 20:24:35.618662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.618652    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.620419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.620392    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.625536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.625508    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgq4s\" (UniqueName: \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\") pod \"maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:35.758078 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:35.758041    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:36.063878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.063832    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"]
Nov 01 20:24:36.073595 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:36.069021    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3607c0_8c57_45dd_b442_10e1fc15245e.slice/crio-7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7 WatchSource:0}: Error finding container 7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7: Status 404 returned error can't find the container with id 7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7
Nov 01 20:24:36.147768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.147731    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:36.149271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.149247    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:36.152160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.152139    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:24:36.716142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.716048    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:24:36.716321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:36.716243    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" containerID="cri-o://75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7" gracePeriod=30
Nov 01 20:24:37.030674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.030001    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s" event={"ID":"4c3607c0-8c57-45dd-b442-10e1fc15245e","Type":"ContainerStarted","Data":"2541f233694f57b9a6062b7349bf9ec51e4d83a187069b9f5adfc8dd7db25e61"}
Nov 01 20:24:37.030674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.030057    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s" event={"ID":"4c3607c0-8c57-45dd-b442-10e1fc15245e","Type":"ContainerStarted","Data":"7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7"}
Nov 01 20:24:37.049041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.048990    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s" podStartSLOduration=2.048976194 podStartE2EDuration="2.048976194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:37.046589043 +0000 UTC m=+1564.767667723" watchObservedRunningTime="2026-11-01 20:24:37.048976194 +0000 UTC m=+1564.770054869"
Nov 01 20:24:37.527441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.527408    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"]
Nov 01 20:24:37.532268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.532236    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.538937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.538906    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"]
Nov 01 20:24:37.584119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.584084    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6gx\" (UniqueName: \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.584292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.584137    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.584292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.584289    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.685949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.685916    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6gx\" (UniqueName: \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.686105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.685965    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.686105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.686048    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.687551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.687518    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.696707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.696676    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.698452 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.698423    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6gx\" (UniqueName: \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\") pod \"maas-api-revoke-sub-f258d1ff-tdmxf\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:37.895283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:37.895235    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:38.058955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:38.058908    5041 generic.go:356] "Generic (PLEG): container finished" podID="4c3607c0-8c57-45dd-b442-10e1fc15245e" containerID="2541f233694f57b9a6062b7349bf9ec51e4d83a187069b9f5adfc8dd7db25e61" exitCode=0
Nov 01 20:24:38.059161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:38.059093    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s" event={"ID":"4c3607c0-8c57-45dd-b442-10e1fc15245e","Type":"ContainerDied","Data":"2541f233694f57b9a6062b7349bf9ec51e4d83a187069b9f5adfc8dd7db25e61"}
Nov 01 20:24:38.180677 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:38.180640    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a945cc4_b103_4b78_9224_96cf9ae37294.slice/crio-1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741 WatchSource:0}: Error finding container 1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741: Status 404 returned error can't find the container with id 1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741
Nov 01 20:24:38.180971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:38.180947    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"]
Nov 01 20:24:39.072039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.071930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf" event={"ID":"4a945cc4-b103-4b78-9224-96cf9ae37294","Type":"ContainerStarted","Data":"91baeb0089b48191bcfa815ad83740a317d4a29b9a128ca78a963addaf009c63"}
Nov 01 20:24:39.072039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.072000    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf" event={"ID":"4a945cc4-b103-4b78-9224-96cf9ae37294","Type":"ContainerStarted","Data":"1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741"}
Nov 01 20:24:39.092468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.092418    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf" podStartSLOduration=2.092404744 podStartE2EDuration="2.092404744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:39.090572954 +0000 UTC m=+1566.811651630" watchObservedRunningTime="2026-11-01 20:24:39.092404744 +0000 UTC m=+1566.813483452"
Nov 01 20:24:39.278874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.278843    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:39.414281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.414164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\") pod \"4c3607c0-8c57-45dd-b442-10e1fc15245e\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") "
Nov 01 20:24:39.414436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.414289    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\") pod \"4c3607c0-8c57-45dd-b442-10e1fc15245e\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") "
Nov 01 20:24:39.414436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.414374    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\" (UniqueName: \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\") pod \"4c3607c0-8c57-45dd-b442-10e1fc15245e\" (UID: \"4c3607c0-8c57-45dd-b442-10e1fc15245e\") "
Nov 01 20:24:39.414721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.414687    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca" pod "4c3607c0-8c57-45dd-b442-10e1fc15245e" (UID: "4c3607c0-8c57-45dd-b442-10e1fc15245e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:39.415061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.415043    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:39.416968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.416940    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca" pod "4c3607c0-8c57-45dd-b442-10e1fc15245e" (UID: "4c3607c0-8c57-45dd-b442-10e1fc15245e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:39.417058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.416968    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s" pod "4c3607c0-8c57-45dd-b442-10e1fc15245e" (UID: "4c3607c0-8c57-45dd-b442-10e1fc15245e"). InnerVolumeSpecName "kube-api-access-jgq4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:39.516480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.516445    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c3607c0-8c57-45dd-b442-10e1fc15245e-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:39.516480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:39.516474    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgq4s\" (UniqueName: \"kubernetes.io/projected/4c3607c0-8c57-45dd-b442-10e1fc15245e-kube-api-access-jgq4s\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:40.084369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:40.084336    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"
Nov 01 20:24:40.084868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:40.084381    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s" event={"ID":"4c3607c0-8c57-45dd-b442-10e1fc15245e","Type":"ContainerDied","Data":"7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7"}
Nov 01 20:24:40.084868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:40.084441    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7978e2169e18648464561a590c2afc07c83af477f3cdc2077c13306a7936d2f7"
Nov 01 20:24:40.086518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:40.086492    5041 generic.go:356] "Generic (PLEG): container finished" podID="4a945cc4-b103-4b78-9224-96cf9ae37294" containerID="91baeb0089b48191bcfa815ad83740a317d4a29b9a128ca78a963addaf009c63" exitCode=0
Nov 01 20:24:40.086640 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:40.086577    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf" event={"ID":"4a945cc4-b103-4b78-9224-96cf9ae37294","Type":"ContainerDied","Data":"91baeb0089b48191bcfa815ad83740a317d4a29b9a128ca78a963addaf009c63"}
Nov 01 20:24:41.320969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.320937    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:41.452729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.452643    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\" (UniqueName: \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\") pod \"4a945cc4-b103-4b78-9224-96cf9ae37294\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") "
Nov 01 20:24:41.452932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.452726    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\") pod \"4a945cc4-b103-4b78-9224-96cf9ae37294\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") "
Nov 01 20:24:41.452932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.452760    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\") pod \"4a945cc4-b103-4b78-9224-96cf9ae37294\" (UID: \"4a945cc4-b103-4b78-9224-96cf9ae37294\") "
Nov 01 20:24:41.453215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.453190    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca" pod "4a945cc4-b103-4b78-9224-96cf9ae37294" (UID: "4a945cc4-b103-4b78-9224-96cf9ae37294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:41.466036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.455150    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:41.466036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.460239    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca" pod "4a945cc4-b103-4b78-9224-96cf9ae37294" (UID: "4a945cc4-b103-4b78-9224-96cf9ae37294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:41.466235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.466103    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx" pod "4a945cc4-b103-4b78-9224-96cf9ae37294" (UID: "4a945cc4-b103-4b78-9224-96cf9ae37294"). InnerVolumeSpecName "kube-api-access-8t6gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:41.533606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.533569    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:24:41.533746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.533632    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:24:41.556650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.556617    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6gx\" (UniqueName: \"kubernetes.io/projected/4a945cc4-b103-4b78-9224-96cf9ae37294-kube-api-access-8t6gx\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:41.556650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:41.556645    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a945cc4-b103-4b78-9224-96cf9ae37294-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.003716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.003688    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:24:42.104153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.104122    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"
Nov 01 20:24:42.104342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.104127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf" event={"ID":"4a945cc4-b103-4b78-9224-96cf9ae37294","Type":"ContainerDied","Data":"1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741"}
Nov 01 20:24:42.104342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.104300    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1078f8cfc174576212fea5e85674472588336770aaa155a801f6347a29da1741"
Nov 01 20:24:42.106026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.105996    5041 generic.go:356] "Generic (PLEG): container finished" podID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerID="75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7" exitCode=0
Nov 01 20:24:42.106152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.106035    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"
Nov 01 20:24:42.106152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.106074    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerDied","Data":"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"}
Nov 01 20:24:42.106152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.106122    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6" event={"ID":"85a9ddc4-339c-4b78-870b-a578d8357f2b","Type":"ContainerDied","Data":"6cd8f9739a7ca4a5321c4cb5bd76fea3c5db4c413e3a4d2c20ed9e29d4d122c1"}
Nov 01 20:24:42.106152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.106150    5041 scope.go:122] "RemoveContainer" containerID="75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"
Nov 01 20:24:42.117842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.117770    5041 scope.go:122] "RemoveContainer" containerID="39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"
Nov 01 20:24:42.128656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.128636    5041 scope.go:122] "RemoveContainer" containerID="75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"
Nov 01 20:24:42.128981 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:42.128959    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7\": container with ID starting with 75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7 not found: ID does not exist" containerID="75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"
Nov 01 20:24:42.129068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.128990    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7"} err="failed to get container status \"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7\": rpc error: code = NotFound desc = could not find container \"75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7\": container with ID starting with 75a7bfa2c2ca111ff9aa44104c9158b5386969dc2b37ccf1c31ab35635931fb7 not found: ID does not exist"
Nov 01 20:24:42.129068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.129007    5041 scope.go:122] "RemoveContainer" containerID="39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"
Nov 01 20:24:42.129362 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:42.129340    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4\": container with ID starting with 39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4 not found: ID does not exist" containerID="39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"
Nov 01 20:24:42.129424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.129370    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4"} err="failed to get container status \"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4\": rpc error: code = NotFound desc = could not find container \"39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4\": container with ID starting with 39b35b7981b3c56bccf55a3af4631e58d68ce2444ff8f68bc505e6f9fb219ca4 not found: ID does not exist"
Nov 01 20:24:42.147770 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.147743    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:24:42.171762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.171737    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.171956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.171838    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.171956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.171880    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.171956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.171918    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.172215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172189    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:42.172342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172272    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.172431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172380    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.172503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172459    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.172586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172569    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.172643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.172608    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\") pod \"85a9ddc4-339c-4b78-870b-a578d8357f2b\" (UID: \"85a9ddc4-339c-4b78-870b-a578d8357f2b\") "
Nov 01 20:24:42.173520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.173346    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.173520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.173390    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:42.173520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.173499    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:42.173741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.173607    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:42.173969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.173421    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:42.175291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.175269    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:24:42.175291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.175279    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:42.175460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.175332    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "kube-api-access-w2v4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:42.175516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.175508    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token" pod "85a9ddc4-339c-4b78-870b-a578d8357f2b" (UID: "85a9ddc4-339c-4b78-870b-a578d8357f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275314    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275347    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a9ddc4-339c-4b78-870b-a578d8357f2b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275357    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275366    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275375    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2v4h\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-kube-api-access-w2v4h\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275384    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275393    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.275397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.275401    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a9ddc4-339c-4b78-870b-a578d8357f2b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:42.439062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.439021    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:24:42.439730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:42.439705    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6f810f-openshift-default-5b8d58b9b9-bbjf6"]
Nov 01 20:24:44.430514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:44.430420    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" path="/var/lib/kubelet/pods/85a9ddc4-339c-4b78-870b-a578d8357f2b/volumes"
Nov 01 20:24:48.675242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675207    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"]
Nov 01 20:24:48.675893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675877    5041 cpu_manager.go:414] "containerMap: removing container" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:48.675893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675891    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:48.676008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675908    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4a945cc4-b103-4b78-9224-96cf9ae37294" containerName="revoke-keys"
Nov 01 20:24:48.676008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675914    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a945cc4-b103-4b78-9224-96cf9ae37294" containerName="revoke-keys"
Nov 01 20:24:48.676008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675962    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4c3607c0-8c57-45dd-b442-10e1fc15245e" containerName="revoke-keys"
Nov 01 20:24:48.676008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.675968    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3607c0-8c57-45dd-b442-10e1fc15245e" containerName="revoke-keys"
Nov 01 20:24:48.676140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.676066    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:48.676140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.676075    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3607c0-8c57-45dd-b442-10e1fc15245e" containerName="revoke-keys"
Nov 01 20:24:48.676140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.676088    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a945cc4-b103-4b78-9224-96cf9ae37294" containerName="revoke-keys"
Nov 01 20:24:48.680926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.680866    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.684364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.684339    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:24:48.685166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.685119    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:24:48.685266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.685202    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:24:48.685412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.685383    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:24:48.687572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.687548    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"]
Nov 01 20:24:48.715012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.714968    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv44q\" (UniqueName: \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.715415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.715391    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.715676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.715656    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.882850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.820865    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv44q\" (UniqueName: \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.882850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.821028    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.882850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.821111    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.882850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.822530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.882850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.826493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:48.921105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:48.921003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv44q\" (UniqueName: \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\") pod \"maas-api-revoke-sub-21d59845-g8zlh\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:49.000845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:49.000428    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:49.314076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:49.314043    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"]
Nov 01 20:24:49.314226 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:49.314144    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab6debb_63fa_43d6_a127_6b1d8e43c1f8.slice/crio-2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa WatchSource:0}: Error finding container 2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa: Status 404 returned error can't find the container with id 2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa
Nov 01 20:24:50.372075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:50.372033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh" event={"ID":"cab6debb-63fa-43d6-a127-6b1d8e43c1f8","Type":"ContainerStarted","Data":"29c096740696ba49871108a822becb9b9ce8f8a83d75ef0b538c1b5afbdeff6c"}
Nov 01 20:24:50.372668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:50.372643    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh" event={"ID":"cab6debb-63fa-43d6-a127-6b1d8e43c1f8","Type":"ContainerStarted","Data":"2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa"}
Nov 01 20:24:50.394865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:50.394811    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh" podStartSLOduration=2.394797136 podStartE2EDuration="2.394797136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:50.393494996 +0000 UTC m=+1578.114573673" watchObservedRunningTime="2026-11-01 20:24:50.394797136 +0000 UTC m=+1578.115875802"
Nov 01 20:24:51.383768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:51.383730    5041 generic.go:356] "Generic (PLEG): container finished" podID="cab6debb-63fa-43d6-a127-6b1d8e43c1f8" containerID="29c096740696ba49871108a822becb9b9ce8f8a83d75ef0b538c1b5afbdeff6c" exitCode=0
Nov 01 20:24:51.384340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:51.383838    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh" event={"ID":"cab6debb-63fa-43d6-a127-6b1d8e43c1f8","Type":"ContainerDied","Data":"29c096740696ba49871108a822becb9b9ce8f8a83d75ef0b538c1b5afbdeff6c"}
Nov 01 20:24:52.569590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.569562    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:52.663335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.663253    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\") pod \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") "
Nov 01 20:24:52.663470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.663438    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\") pod \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") "
Nov 01 20:24:52.663508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.663472    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\" (UniqueName: \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\") pod \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\" (UID: \"cab6debb-63fa-43d6-a127-6b1d8e43c1f8\") "
Nov 01 20:24:52.663810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.663766    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca" pod "cab6debb-63fa-43d6-a127-6b1d8e43c1f8" (UID: "cab6debb-63fa-43d6-a127-6b1d8e43c1f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:52.664147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.664126    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:52.665334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.665307    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca" pod "cab6debb-63fa-43d6-a127-6b1d8e43c1f8" (UID: "cab6debb-63fa-43d6-a127-6b1d8e43c1f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:52.665671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.665650    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q" pod "cab6debb-63fa-43d6-a127-6b1d8e43c1f8" (UID: "cab6debb-63fa-43d6-a127-6b1d8e43c1f8"). InnerVolumeSpecName "kube-api-access-pv44q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:52.766032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.765999    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv44q\" (UniqueName: \"kubernetes.io/projected/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-kube-api-access-pv44q\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:52.766032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:52.766031    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab6debb-63fa-43d6-a127-6b1d8e43c1f8-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:53.400982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:53.400950    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"
Nov 01 20:24:53.401177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:53.400948    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh" event={"ID":"cab6debb-63fa-43d6-a127-6b1d8e43c1f8","Type":"ContainerDied","Data":"2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa"}
Nov 01 20:24:53.401177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:53.401089    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2464290e42451ae00e93746728da2a175ec3560026bf5210a2915d888e44ffaa"
Nov 01 20:24:54.024820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.024767    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"]
Nov 01 20:24:54.029109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.029081    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b276f2e3-kddx4"]
Nov 01 20:24:54.343940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.343905    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"]
Nov 01 20:24:54.344606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344590    5041 cpu_manager.go:414] "containerMap: removing container" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:54.344606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344605    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:54.344797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344636    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cab6debb-63fa-43d6-a127-6b1d8e43c1f8" containerName="revoke-keys"
Nov 01 20:24:54.344797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344641    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab6debb-63fa-43d6-a127-6b1d8e43c1f8" containerName="revoke-keys"
Nov 01 20:24:54.344797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344749    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab6debb-63fa-43d6-a127-6b1d8e43c1f8" containerName="revoke-keys"
Nov 01 20:24:54.344797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.344760    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a9ddc4-339c-4b78-870b-a578d8357f2b" containerName="istio-proxy"
Nov 01 20:24:54.348756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.348731    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.353583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.353559    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:24:54.354048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.354022    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:24:54.354187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.354107    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:24:54.356972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.356946    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"]
Nov 01 20:24:54.360133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.360115    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:24:54.392108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.392072    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.392291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.392269    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn45h\" (UniqueName: \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.392357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.392315    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.430134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.430102    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b591ff86-68e6-4d25-9fa3-2baafd863077" path="/var/lib/kubelet/pods/b591ff86-68e6-4d25-9fa3-2baafd863077/volumes"
Nov 01 20:24:54.493926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.493882    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn45h\" (UniqueName: \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.494121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.493966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.494399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.494373    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.495167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.495141    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.497237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.497217    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.506614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.506588    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn45h\" (UniqueName: \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\") pod \"maas-api-revoke-sub-54dc5816-mb6kd\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.668154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.668063    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:54.959363 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:24:54.959328    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6432505_25f6_4a47_b482_4b7a2d64d582.slice/crio-c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28 WatchSource:0}: Error finding container c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28: Status 404 returned error can't find the container with id c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28
Nov 01 20:24:54.959363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:54.959353    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"]
Nov 01 20:24:55.022411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:55.022380    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"]
Nov 01 20:24:55.026754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:55.026731    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07be0125-rq8gt"]
Nov 01 20:24:55.423637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:55.423599    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd" event={"ID":"e6432505-25f6-4a47-b482-4b7a2d64d582","Type":"ContainerStarted","Data":"d90bb41ea2f53cfcb8c92a0c33807986da8495e23044eac65c31697fed7ae213"}
Nov 01 20:24:55.423849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:55.423659    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd" event={"ID":"e6432505-25f6-4a47-b482-4b7a2d64d582","Type":"ContainerStarted","Data":"c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28"}
Nov 01 20:24:55.440276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:55.440226    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd" podStartSLOduration=1.440213052 podStartE2EDuration="1.440213052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:55.437890765 +0000 UTC m=+1583.158969442" watchObservedRunningTime="2026-11-01 20:24:55.440213052 +0000 UTC m=+1583.161291728"
Nov 01 20:24:56.430460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:56.430426    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b4fa09-c1da-4350-9efe-e06a2dc06f84" path="/var/lib/kubelet/pods/f1b4fa09-c1da-4350-9efe-e06a2dc06f84/volumes"
Nov 01 20:24:56.434298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:56.434272    5041 generic.go:356] "Generic (PLEG): container finished" podID="e6432505-25f6-4a47-b482-4b7a2d64d582" containerID="d90bb41ea2f53cfcb8c92a0c33807986da8495e23044eac65c31697fed7ae213" exitCode=0
Nov 01 20:24:56.434419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:56.434349    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd" event={"ID":"e6432505-25f6-4a47-b482-4b7a2d64d582","Type":"ContainerDied","Data":"d90bb41ea2f53cfcb8c92a0c33807986da8495e23044eac65c31697fed7ae213"}
Nov 01 20:24:57.183059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.183016    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" containerID="cri-o://a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb" gracePeriod=45
Nov 01 20:24:57.264180 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:57.264140    5041 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919c5973_9291_4d84_bdda_5f203248be84.slice/crio-conmon-a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod919c5973_9291_4d84_bdda_5f203248be84.slice/crio-a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 20:24:57.335044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.334997    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused"
Nov 01 20:24:57.444304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.444227    5041 generic.go:356] "Generic (PLEG): container finished" podID="919c5973-9291-4d84-bdda-5f203248be84" containerID="a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb" exitCode=0
Nov 01 20:24:57.444621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.444296    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" event={"ID":"919c5973-9291-4d84-bdda-5f203248be84","Type":"ContainerDied","Data":"a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb"}
Nov 01 20:24:57.444621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.444338    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd" event={"ID":"919c5973-9291-4d84-bdda-5f203248be84","Type":"ContainerDied","Data":"93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea"}
Nov 01 20:24:57.444621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.444353    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93a5da35383b30f1b450986940666d17a5a22d5112c4c8443477be58058a44ea"
Nov 01 20:24:57.470564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.470122    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:24:57.550057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550016    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\" (UniqueName: \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550129    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550187    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550299    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550328    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550440    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\" (UniqueName: \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\") pod \"919c5973-9291-4d84-bdda-5f203248be84\" (UID: \"919c5973-9291-4d84-bdda-5f203248be84\") "
Nov 01 20:24:57.550488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550453    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.550661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.550499    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.551088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.551064    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.551256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.551204    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.551382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.551365    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.553628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.553596    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:57.553738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.553713    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "kube-api-access-fsggs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:57.554434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.554408    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm" pod "919c5973-9291-4d84-bdda-5f203248be84" (UID: "919c5973-9291-4d84-bdda-5f203248be84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:24:57.652713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.652680    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919c5973-9291-4d84-bdda-5f203248be84-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.652713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.652714    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.653012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.652728    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919c5973-9291-4d84-bdda-5f203248be84-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.653012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.652744    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsggs\" (UniqueName: \"kubernetes.io/projected/919c5973-9291-4d84-bdda-5f203248be84-kube-api-access-fsggs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.656217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.655885    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:57.778949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.778728    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\" (UniqueName: \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\") pod \"e6432505-25f6-4a47-b482-4b7a2d64d582\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") "
Nov 01 20:24:57.778949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.778806    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\") pod \"e6432505-25f6-4a47-b482-4b7a2d64d582\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") "
Nov 01 20:24:57.779908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.779363    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\") pod \"e6432505-25f6-4a47-b482-4b7a2d64d582\" (UID: \"e6432505-25f6-4a47-b482-4b7a2d64d582\") "
Nov 01 20:24:57.794049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.792193    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca" pod "e6432505-25f6-4a47-b482-4b7a2d64d582" (UID: "e6432505-25f6-4a47-b482-4b7a2d64d582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:24:57.794049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.792281    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca" pod "e6432505-25f6-4a47-b482-4b7a2d64d582" (UID: "e6432505-25f6-4a47-b482-4b7a2d64d582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:24:57.802285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.802227    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h" pod "e6432505-25f6-4a47-b482-4b7a2d64d582" (UID: "e6432505-25f6-4a47-b482-4b7a2d64d582"). InnerVolumeSpecName "kube-api-access-nn45h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:24:57.883651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.883620    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn45h\" (UniqueName: \"kubernetes.io/projected/e6432505-25f6-4a47-b482-4b7a2d64d582-kube-api-access-nn45h\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.884010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.883992    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:57.884174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:57.884163    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6432505-25f6-4a47-b482-4b7a2d64d582-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:24:58.453931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.453905    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"
Nov 01 20:24:58.454360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.453936    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"
Nov 01 20:24:58.454360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.453943    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd" event={"ID":"e6432505-25f6-4a47-b482-4b7a2d64d582","Type":"ContainerDied","Data":"c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28"}
Nov 01 20:24:58.454360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.453995    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1bd410df4e328b81f14f7ace069cd80bb894fde15ebb4e8ced4128d4a7f9b28"
Nov 01 20:24:58.482265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.482144    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:24:58.486335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:24:58.486305    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/premium-w2-e27ae7-kserve-6dd5755d97-7wpsd"]
Nov 01 20:24:58.542953 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:24:58.542916    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:25:00.430300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:00.430264    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919c5973-9291-4d84-bdda-5f203248be84" path="/var/lib/kubelet/pods/919c5973-9291-4d84-bdda-5f203248be84/volumes"
Nov 01 20:25:05.038491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:05.038449    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"]
Nov 01 20:25:05.045215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:05.045182    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b11a20-qjtw6"]
Nov 01 20:25:06.451491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:06.451358    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd55686-ab0f-4d04-bed7-f4adf1f12445" path="/var/lib/kubelet/pods/7bd55686-ab0f-4d04-bed7-f4adf1f12445/volumes"
Nov 01 20:25:07.022761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.022722    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"]
Nov 01 20:25:07.026954 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.026928    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29d210a-zj5cj"]
Nov 01 20:25:07.110123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.110086    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"]
Nov 01 20:25:07.114807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.111430    5041 cpu_manager.go:414] "containerMap: removing container" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main"
Nov 01 20:25:07.124143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.124116    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main"
Nov 01 20:25:07.124355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.124340    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e6432505-25f6-4a47-b482-4b7a2d64d582" containerName="revoke-keys"
Nov 01 20:25:07.124450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.124438    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6432505-25f6-4a47-b482-4b7a2d64d582" containerName="revoke-keys"
Nov 01 20:25:07.124847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.124826    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="919c5973-9291-4d84-bdda-5f203248be84" containerName="main"
Nov 01 20:25:07.124987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.124974    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6432505-25f6-4a47-b482-4b7a2d64d582" containerName="revoke-keys"
Nov 01 20:25:07.150975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.150946    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.155816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.155768    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:25:07.156155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.156074    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:25:07.156446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.156430    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:25:07.156593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.156580    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:25:07.158207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.158185    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"]
Nov 01 20:25:07.235112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.215744    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.235112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.215882    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsclb\" (UniqueName: \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.235112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.216078    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.317319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.317285    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.317515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.317386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsclb\" (UniqueName: \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.317591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.317572    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.318490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.318460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.320479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.320445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.339444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.339410    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsclb\" (UniqueName: \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\") pod \"maas-api-revoke-sub-0bb9c6ec-dq6bf\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.610603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.610512    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:07.909059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:07.909025    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"]
Nov 01 20:25:07.909464 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:25:07.909442    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc737061_a9d4_4d3f_8172_dd05a993fc9b.slice/crio-93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e WatchSource:0}: Error finding container 93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e: Status 404 returned error can't find the container with id 93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e
Nov 01 20:25:08.432063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:08.432027    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae" path="/var/lib/kubelet/pods/9da1cc8c-94a4-4d6c-8fe5-93e267cb81ae/volumes"
Nov 01 20:25:08.557492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:08.557452    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf" event={"ID":"bc737061-a9d4-4d3f-8172-dd05a993fc9b","Type":"ContainerStarted","Data":"aaba946aa5dc99ed52f88f540862a0c9b962d18a958b41266c5cd093a979a3d3"}
Nov 01 20:25:08.557682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:08.557524    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf" event={"ID":"bc737061-a9d4-4d3f-8172-dd05a993fc9b","Type":"ContainerStarted","Data":"93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e"}
Nov 01 20:25:08.574261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:08.574199    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf" podStartSLOduration=1.5741799250000001 podStartE2EDuration="1.574179925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:08.573586375 +0000 UTC m=+1596.294665052" watchObservedRunningTime="2026-11-01 20:25:08.574179925 +0000 UTC m=+1596.295258603"
Nov 01 20:25:09.573799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:09.573141    5041 generic.go:356] "Generic (PLEG): container finished" podID="bc737061-a9d4-4d3f-8172-dd05a993fc9b" containerID="aaba946aa5dc99ed52f88f540862a0c9b962d18a958b41266c5cd093a979a3d3" exitCode=0
Nov 01 20:25:09.573799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:09.573342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf" event={"ID":"bc737061-a9d4-4d3f-8172-dd05a993fc9b","Type":"ContainerDied","Data":"aaba946aa5dc99ed52f88f540862a0c9b962d18a958b41266c5cd093a979a3d3"}
Nov 01 20:25:11.383989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.383955    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"]
Nov 01 20:25:11.401632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.401559    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2babaef6-fcb2s"]
Nov 01 20:25:11.442098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.441739    5041 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/health\": context deadline exceeded"
Nov 01 20:25:11.756512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.756470    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf" event={"ID":"bc737061-a9d4-4d3f-8172-dd05a993fc9b","Type":"ContainerDied","Data":"93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e"}
Nov 01 20:25:11.756720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.756544    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f8cd8cce223adbd4d18b1e7cb8d9c5bc1e87c31516f3b2575a347a02a69f9e"
Nov 01 20:25:11.792992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.792953    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:11.800809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.799830    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\") pod \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") "
Nov 01 20:25:11.800809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.799936    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\" (UniqueName: \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\") pod \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") "
Nov 01 20:25:11.800809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.800152    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\") pod \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\" (UID: \"bc737061-a9d4-4d3f-8172-dd05a993fc9b\") "
Nov 01 20:25:11.802541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.801175    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca" pod "bc737061-a9d4-4d3f-8172-dd05a993fc9b" (UID: "bc737061-a9d4-4d3f-8172-dd05a993fc9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:11.803532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.802969    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb" pod "bc737061-a9d4-4d3f-8172-dd05a993fc9b" (UID: "bc737061-a9d4-4d3f-8172-dd05a993fc9b"). InnerVolumeSpecName "kube-api-access-jsclb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:11.803532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.803446    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca" pod "bc737061-a9d4-4d3f-8172-dd05a993fc9b" (UID: "bc737061-a9d4-4d3f-8172-dd05a993fc9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:25:11.804214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.804189    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:11.804355 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.804338    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc737061-a9d4-4d3f-8172-dd05a993fc9b-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:11.804465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:11.804452    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsclb\" (UniqueName: \"kubernetes.io/projected/bc737061-a9d4-4d3f-8172-dd05a993fc9b-kube-api-access-jsclb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:12.433434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:12.433397    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3607c0-8c57-45dd-b442-10e1fc15245e" path="/var/lib/kubelet/pods/4c3607c0-8c57-45dd-b442-10e1fc15245e/volumes"
Nov 01 20:25:12.765282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:12.765203    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"
Nov 01 20:25:21.349529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.348762    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:25:21.352906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.350877    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy" containerID="cri-o://c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4" gracePeriod=30
Nov 01 20:25:21.368709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.368672    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:25:21.369060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.369013    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" containerID="cri-o://489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a" gracePeriod=30
Nov 01 20:25:21.377669 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.377641    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:25:21.378067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.378039    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" containerID="cri-o://66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad" gracePeriod=30
Nov 01 20:25:21.383012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.382932    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:25:21.383259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.383181    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy" containerID="cri-o://20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b" gracePeriod=30
Nov 01 20:25:21.694890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.694763    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:25:21.694890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:21.694873    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:25:23.123535 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:23.123498    5041 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:25:23.124033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:23.123553    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:25:26.765185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.765156    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:25:26.837180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.837149    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:25:26.841070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.841042    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.886766    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.886891    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.886921    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887000    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887080    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887154    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887217    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887261    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887363    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887406    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887487    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887537    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887587    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887626    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887663    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\") pod \"2270b821-22ea-4bb2-bc5e-7bcea6468704\" (UID: \"2270b821-22ea-4bb2-bc5e-7bcea6468704\") "
Nov 01 20:25:26.887827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.887717    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\") pod \"54b6a89d-9b18-45fd-a952-3fe23382defa\" (UID: \"54b6a89d-9b18-45fd-a952-3fe23382defa\") "
Nov 01 20:25:26.889167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.888682    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.889167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.888907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.890309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.890265    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.891075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.890533    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:26.891075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.890873    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:26.891223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.891195    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.891416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.891399    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.892150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.892095    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:26.893760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.893510    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "kube-api-access-6cdw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:26.894076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.893959    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.894172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894104    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.894229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894195    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.894304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894280    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:26.894417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894351    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.894484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894410    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.894484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894407    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token" pod "54b6a89d-9b18-45fd-a952-3fe23382defa" (UID: "54b6a89d-9b18-45fd-a952-3fe23382defa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:26.894484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.894459    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:26.896124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.896101    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9" pod "2270b821-22ea-4bb2-bc5e-7bcea6468704" (UID: "2270b821-22ea-4bb2-bc5e-7bcea6468704"). InnerVolumeSpecName "kube-api-access-6cxs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:26.988931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.988885    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.989121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989041    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.989188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989156    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.989259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989200    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.989259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989251    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.989366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989289    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989497    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989553    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.989583    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\") pod \"44650ec9-6c47-4dea-9f35-d807985ab580\" (UID: \"44650ec9-6c47-4dea-9f35-d807985ab580\") "
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.990244    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.990469    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.990985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.990668    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.991565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991025    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:26.991938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991909    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991942    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdw9\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-kube-api-access-6cdw9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991958    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991975    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.991989    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992003    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992017    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992030    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992044    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992057    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992071    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992086    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxs9\" (UniqueName: \"kubernetes.io/projected/2270b821-22ea-4bb2-bc5e-7bcea6468704-kube-api-access-6cxs9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992149    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992163    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992177    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b6a89d-9b18-45fd-a952-3fe23382defa-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992191    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b6a89d-9b18-45fd-a952-3fe23382defa-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992214    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992227    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992243    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44650ec9-6c47-4dea-9f35-d807985ab580-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992257    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2270b821-22ea-4bb2-bc5e-7bcea6468704-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992270    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2270b821-22ea-4bb2-bc5e-7bcea6468704-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.992488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.992285    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b6a89d-9b18-45fd-a952-3fe23382defa-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:26.993382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.993053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.994652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.994629    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "kube-api-access-cfjjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:26.995434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.995405    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:26.996371 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.996344    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:26.996447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:26.996383    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token" pod "44650ec9-6c47-4dea-9f35-d807985ab580" (UID: "44650ec9-6c47-4dea-9f35-d807985ab580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.049301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.049265    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:25:27.077568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.077527    5041 generic.go:356] "Generic (PLEG): container finished" podID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerID="66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad" exitCode=0
Nov 01 20:25:27.077762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.077585    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"
Nov 01 20:25:27.077762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.077644    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" event={"ID":"54b6a89d-9b18-45fd-a952-3fe23382defa","Type":"ContainerDied","Data":"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"}
Nov 01 20:25:27.077762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.077717    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd" event={"ID":"54b6a89d-9b18-45fd-a952-3fe23382defa","Type":"ContainerDied","Data":"807e7943dd0a59b833a3c3a785dd19ef52a7b4089f0220b9065b7c6df84b3902"}
Nov 01 20:25:27.078106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.077770    5041 scope.go:122] "RemoveContainer" containerID="66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"
Nov 01 20:25:27.088687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.088652    5041 generic.go:356] "Generic (PLEG): container finished" podID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerID="c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4" exitCode=0
Nov 01 20:25:27.088900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.088772    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" event={"ID":"a05fc0dc-efeb-4b19-a66a-9e3431c87052","Type":"ContainerDied","Data":"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"}
Nov 01 20:25:27.089009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.088939    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g" event={"ID":"a05fc0dc-efeb-4b19-a66a-9e3431c87052","Type":"ContainerDied","Data":"2f0ecde50593bbf4712e4fc5c105cdefa954a0357c11bed92fe8e2b7aed7dd8b"}
Nov 01 20:25:27.089890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.089868    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"
Nov 01 20:25:27.091764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.091727    5041 generic.go:356] "Generic (PLEG): container finished" podID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerID="20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b" exitCode=0
Nov 01 20:25:27.091950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.091924    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"
Nov 01 20:25:27.092604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.092110    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" event={"ID":"2270b821-22ea-4bb2-bc5e-7bcea6468704","Type":"ContainerDied","Data":"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"}
Nov 01 20:25:27.092604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.092168    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x" event={"ID":"2270b821-22ea-4bb2-bc5e-7bcea6468704","Type":"ContainerDied","Data":"33ae15e1b0543e02a43ae888c9487f3e5c8f99c1ef0ec55d4a2778b2dfadcc13"}
Nov 01 20:25:27.093831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.093618    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.093831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.093646    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44650ec9-6c47-4dea-9f35-d807985ab580-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.093831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.093660    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.093831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.093675    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44650ec9-6c47-4dea-9f35-d807985ab580-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.093831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.093689    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfjjd\" (UniqueName: \"kubernetes.io/projected/44650ec9-6c47-4dea-9f35-d807985ab580-kube-api-access-cfjjd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.095494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.095465    5041 generic.go:356] "Generic (PLEG): container finished" podID="44650ec9-6c47-4dea-9f35-d807985ab580" containerID="489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a" exitCode=0
Nov 01 20:25:27.095729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.095708    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" event={"ID":"44650ec9-6c47-4dea-9f35-d807985ab580","Type":"ContainerDied","Data":"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"}
Nov 01 20:25:27.095854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.095755    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm" event={"ID":"44650ec9-6c47-4dea-9f35-d807985ab580","Type":"ContainerDied","Data":"e9a58a26acb1a069195e3b5824f527d11b4c1ba8c256fd8d58037acf107f3158"}
Nov 01 20:25:27.095854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.095835    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"
Nov 01 20:25:27.117033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.117006    5041 scope.go:122] "RemoveContainer" containerID="66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"
Nov 01 20:25:27.117296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.117125    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:25:27.117525 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:27.117491    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad\": container with ID starting with 66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad not found: ID does not exist" containerID="66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"
Nov 01 20:25:27.117650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.117530    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad"} err="failed to get container status \"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad\": rpc error: code = NotFound desc = could not find container \"66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad\": container with ID starting with 66c09ac9558f1c17439bb9b3b4ce4a0625859bd50ce971a2c24ddcec3803b2ad not found: ID does not exist"
Nov 01 20:25:27.117650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.117555    5041 scope.go:122] "RemoveContainer" containerID="c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"
Nov 01 20:25:27.122486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.122466    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-zrjrd"]
Nov 01 20:25:27.132362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.132340    5041 scope.go:122] "RemoveContainer" containerID="c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"
Nov 01 20:25:27.132724 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:27.132695    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4\": container with ID starting with c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4 not found: ID does not exist" containerID="c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"
Nov 01 20:25:27.132920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.132887    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4"} err="failed to get container status \"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4\": rpc error: code = NotFound desc = could not find container \"c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4\": container with ID starting with c760a5d3ec0e0d29fd4ed50299914e5589a5e77bbe86a6c649b18e6bb5e143f4 not found: ID does not exist"
Nov 01 20:25:27.133015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.132922    5041 scope.go:122] "RemoveContainer" containerID="20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"
Nov 01 20:25:27.138600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.138577    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:25:27.145815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.145774    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-q8chm"]
Nov 01 20:25:27.154170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.154141    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:25:27.157153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.157125    5041 scope.go:122] "RemoveContainer" containerID="20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"
Nov 01 20:25:27.157506 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:27.157484    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b\": container with ID starting with 20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b not found: ID does not exist" containerID="20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"
Nov 01 20:25:27.157578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.157514    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b"} err="failed to get container status \"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b\": rpc error: code = NotFound desc = could not find container \"20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b\": container with ID starting with 20af2e9f870e5f38e7cacc283b8246d41588a04c1e25defd1297e6169fbe800b not found: ID does not exist"
Nov 01 20:25:27.157578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.157534    5041 scope.go:122] "RemoveContainer" containerID="489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"
Nov 01 20:25:27.158851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.158830    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bsb6x"]
Nov 01 20:25:27.168515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.168491    5041 scope.go:122] "RemoveContainer" containerID="489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"
Nov 01 20:25:27.168853 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:27.168832    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a\": container with ID starting with 489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a not found: ID does not exist" containerID="489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"
Nov 01 20:25:27.168917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.168864    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a"} err="failed to get container status \"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a\": rpc error: code = NotFound desc = could not find container \"489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a\": container with ID starting with 489ab17985f5516053e9fdf2c76594df6c64b313e34f8432712948d626e3193a not found: ID does not exist"
Nov 01 20:25:27.194187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194150    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194220    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194265    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194457    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194505    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194563    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.194647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194607    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:27.194647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194618    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194663    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194709    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.194898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.194760    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\") pod \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\" (UID: \"a05fc0dc-efeb-4b19-a66a-9e3431c87052\") "
Nov 01 20:25:27.195079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195043    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.195079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195060    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.195284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195264    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.195540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195507    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.195540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195536    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.195653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195553    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.195653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195569    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.195653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.195587    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.196745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.196719    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:27.197096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.197074    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "kube-api-access-c888f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.197254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.197232    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:27.197356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.197257    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token" pod "a05fc0dc-efeb-4b19-a66a-9e3431c87052" (UID: "a05fc0dc-efeb-4b19-a66a-9e3431c87052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:27.296791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.296740    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.296960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.296773    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.296960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.296823    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05fc0dc-efeb-4b19-a66a-9e3431c87052-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.296960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.296836    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c888f\" (UniqueName: \"kubernetes.io/projected/a05fc0dc-efeb-4b19-a66a-9e3431c87052-kube-api-access-c888f\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:27.465917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.465880    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:25:27.470957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:27.470928    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-2vx4g"]
Nov 01 20:25:28.430006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:28.429968    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" path="/var/lib/kubelet/pods/2270b821-22ea-4bb2-bc5e-7bcea6468704/volumes"
Nov 01 20:25:28.430583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:28.430567    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" path="/var/lib/kubelet/pods/44650ec9-6c47-4dea-9f35-d807985ab580/volumes"
Nov 01 20:25:28.431077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:28.431062    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" path="/var/lib/kubelet/pods/54b6a89d-9b18-45fd-a952-3fe23382defa/volumes"
Nov 01 20:25:28.431845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:28.431830    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" path="/var/lib/kubelet/pods/a05fc0dc-efeb-4b19-a66a-9e3431c87052/volumes"
Nov 01 20:25:28.551311 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:28.551265    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:25:35.733085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:35.732969    5041 scope.go:122] "RemoveContainer" containerID="a98ea936375a83ad4a247d098888766a8490a24cccfcd25dc11d02f2ac0791eb"
Nov 01 20:25:40.239921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:40.239873    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Nov 01 20:25:40.241994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:40.240504    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:40.326919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:40.326870    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerName="main" probeResult="failure" output="Get \"https://10.132.0.201:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:25:41.063713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:41.063679    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"]
Nov 01 20:25:41.068936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:41.068439    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fba7223-bjfsl"]
Nov 01 20:25:42.496993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:42.496959    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3a1957-093b-46e1-af08-ece65b0e04ad" path="/var/lib/kubelet/pods/7d3a1957-093b-46e1-af08-ece65b0e04ad/volumes"
Nov 01 20:25:42.525595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:42.525562    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:25:42.525859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:42.525831    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy" containerID="cri-o://30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b" gracePeriod=30
Nov 01 20:25:42.751997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:42.751903    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:25:42.752181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:42.752125    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy" containerID="cri-o://33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d" gracePeriod=30
Nov 01 20:25:47.893793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:47.893750    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:25:48.019321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.019283    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.019591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.019573    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.019745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.019732    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020022    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020137    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020257    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020490    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020622    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.020847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.020833    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\") pod \"8ee4d00a-54c1-4315-a77b-9a81de063dae\" (UID: \"8ee4d00a-54c1-4315-a77b-9a81de063dae\") "
Nov 01 20:25:48.021862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.021838    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.023341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.022897    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.023341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.023081    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.027800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.023721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:48.028001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.027963    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.033907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.033869    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:48.039797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.034165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.043576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.040965    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "kube-api-access-4fpm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:48.044104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.044079    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo" pod "8ee4d00a-54c1-4315-a77b-9a81de063dae" (UID: "8ee4d00a-54c1-4315-a77b-9a81de063dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:48.085497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.085469    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:25:48.123520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123429    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee4d00a-54c1-4315-a77b-9a81de063dae-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123470    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpm6\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-kube-api-access-4fpm6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123486    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123499    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123513    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123529    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123543    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123556    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.123844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.123571    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee4d00a-54c1-4315-a77b-9a81de063dae-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.244004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.243955    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244044    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244099    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244133    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244293    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244378    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244446    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244496    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.244652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.244533    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\") pod \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\" (UID: \"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2\") "
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.248645    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.248861    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.249245    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.252866    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.253591    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.253595    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.253637    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "kube-api-access-7vg9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:25:48.256031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.255993    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:25:48.256651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.256616    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data" pod "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" (UID: "3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:25:48.348185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348147    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348184    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348196    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vg9t\" (UniqueName: \"kubernetes.io/projected/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-kube-api-access-7vg9t\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348213    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348225    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348236    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348249    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348262    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.348422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.348275    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:25:48.606140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.606103    5041 generic.go:356] "Generic (PLEG): container finished" podID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerID="30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b" exitCode=0
Nov 01 20:25:48.606336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.606159    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"
Nov 01 20:25:48.606336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.606194    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" event={"ID":"8ee4d00a-54c1-4315-a77b-9a81de063dae","Type":"ContainerDied","Data":"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"}
Nov 01 20:25:48.606336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.606247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6" event={"ID":"8ee4d00a-54c1-4315-a77b-9a81de063dae","Type":"ContainerDied","Data":"62881c9c6db1828370ad477980996a3ecdab3fe8f66c9d54ce67541008477bf8"}
Nov 01 20:25:48.606336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.606268    5041 scope.go:122] "RemoveContainer" containerID="30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"
Nov 01 20:25:48.607948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.607913    5041 generic.go:356] "Generic (PLEG): container finished" podID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerID="33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d" exitCode=0
Nov 01 20:25:48.608063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.607961    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"
Nov 01 20:25:48.608063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.607980    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" event={"ID":"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2","Type":"ContainerDied","Data":"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"}
Nov 01 20:25:48.608063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.608022    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2" event={"ID":"3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2","Type":"ContainerDied","Data":"5d1e68c43febcc98d6e71c65a136fb5f9924816a1c262cd7a858b94860921c50"}
Nov 01 20:25:48.618028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.618002    5041 scope.go:122] "RemoveContainer" containerID="30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"
Nov 01 20:25:48.618310 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:48.618285    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b\": container with ID starting with 30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b not found: ID does not exist" containerID="30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"
Nov 01 20:25:48.618402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.618324    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b"} err="failed to get container status \"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b\": rpc error: code = NotFound desc = could not find container \"30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b\": container with ID starting with 30cca4f4a3056379c3f6143de045ca73f9c555db21d8bbfdf32ec84e6d98134b not found: ID does not exist"
Nov 01 20:25:48.618402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.618346    5041 scope.go:122] "RemoveContainer" containerID="33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"
Nov 01 20:25:48.629305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.629281    5041 scope.go:122] "RemoveContainer" containerID="33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"
Nov 01 20:25:48.629643 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:48.629622    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d\": container with ID starting with 33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d not found: ID does not exist" containerID="33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"
Nov 01 20:25:48.629736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.629661    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d"} err="failed to get container status \"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d\": rpc error: code = NotFound desc = could not find container \"33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d\": container with ID starting with 33f6a9f944569945e155fefbb3192a117effe76d71e198aabf6a2d25942f331d not found: ID does not exist"
Nov 01 20:25:48.629830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.629751    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:25:48.634601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.634579    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4tbw6"]
Nov 01 20:25:48.645394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.645370    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:25:48.649222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:48.649195    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-9sgs2"]
Nov 01 20:25:50.429130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:50.429099    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" path="/var/lib/kubelet/pods/3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2/volumes"
Nov 01 20:25:50.429678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:50.429661    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" path="/var/lib/kubelet/pods/8ee4d00a-54c1-4315-a77b-9a81de063dae/volumes"
Nov 01 20:25:54.023962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:54.023924    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"]
Nov 01 20:25:54.027637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:54.027613    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d5051e-gtbmt"]
Nov 01 20:25:54.449403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:25:54.449366    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b264aa24-3a85-4f00-b6ce-024bb2ad1dc3" path="/var/lib/kubelet/pods/b264aa24-3a85-4f00-b6ce-024bb2ad1dc3/volumes"
Nov 01 20:25:58.542307 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:25:58.542273    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:26:04.025188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:04.025145    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"]
Nov 01 20:26:04.030024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:04.029988    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43bcb3ef-qc5hb"]
Nov 01 20:26:04.428670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:04.428622    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ca2714-5d58-4cd0-836d-b9b8ccdd2be5" path="/var/lib/kubelet/pods/21ca2714-5d58-4cd0-836d-b9b8ccdd2be5/volumes"
Nov 01 20:26:08.917116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917081    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"]
Nov 01 20:26:08.917734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917719    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917735    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917746    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc737061-a9d4-4d3f-8172-dd05a993fc9b" containerName="revoke-keys"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917751    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc737061-a9d4-4d3f-8172-dd05a993fc9b" containerName="revoke-keys"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917762    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917767    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917807    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy"
Nov 01 20:26:08.917816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917812    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917827    5041 cpu_manager.go:414] "containerMap: removing container" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917832    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917837    5041 cpu_manager.go:414] "containerMap: removing container" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917841    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917848    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917853    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917960    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec88d3f-2ec4-46f7-9740-8bad25ce1ba2" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917970    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b6a89d-9b18-45fd-a952-3fe23382defa" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917977    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="44650ec9-6c47-4dea-9f35-d807985ab580" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917982    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee4d00a-54c1-4315-a77b-9a81de063dae" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917991    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05fc0dc-efeb-4b19-a66a-9e3431c87052" containerName="istio-proxy"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.917997    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc737061-a9d4-4d3f-8172-dd05a993fc9b" containerName="revoke-keys"
Nov 01 20:26:08.918064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.918003    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2270b821-22ea-4bb2-bc5e-7bcea6468704" containerName="istio-proxy"
Nov 01 20:26:08.922878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.922859    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:08.927222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.927195    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:26:08.928434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.928411    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:26:08.928570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.928494    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:26:08.928667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.928537    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:26:08.930211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:08.930182    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"]
Nov 01 20:26:09.082229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.082194    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.082442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.082242    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.082528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.082434    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlg66\" (UniqueName: \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.184333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.184245    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.184333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.184307    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.184530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.184443    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlg66\" (UniqueName: \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.185103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.185079    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.186927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.186900    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.191983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.191961    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlg66\" (UniqueName: \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\") pod \"maas-api-revoke-sub-f1ff52b4-w4dzq\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.243878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.243846    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:09.570292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.570262    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"]
Nov 01 20:26:09.571191 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:26:09.571160    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6972e9_34c1_4cf8_8c10_ad5f3c2cde90.slice/crio-9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907 WatchSource:0}: Error finding container 9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907: Status 404 returned error can't find the container with id 9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907
Nov 01 20:26:09.808292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:09.808255    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq" event={"ID":"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90","Type":"ContainerStarted","Data":"9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907"}
Nov 01 20:26:10.818125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:10.818035    5041 generic.go:356] "Generic (PLEG): container finished" podID="bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" containerID="d71f8b4b660dcc97ccc4c1b4f79c9fa6309c52b366222903e5504ee7805e748a" exitCode=0
Nov 01 20:26:10.818492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:10.818126    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq" event={"ID":"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90","Type":"ContainerDied","Data":"d71f8b4b660dcc97ccc4c1b4f79c9fa6309c52b366222903e5504ee7805e748a"}
Nov 01 20:26:11.996271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:11.996245    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:12.021749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.021709    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\" (UniqueName: \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\") pod \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") "
Nov 01 20:26:12.021974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.021837    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\") pod \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") "
Nov 01 20:26:12.021974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.021920    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\") pod \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\" (UID: \"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90\") "
Nov 01 20:26:12.022641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.022609    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca" pod "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" (UID: "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:12.024550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.024523    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca" pod "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" (UID: "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:12.024969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.024932    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66" pod "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" (UID: "bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90"). InnerVolumeSpecName "kube-api-access-rlg66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:12.124446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.124366    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:12.124446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.124402    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:12.124446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.124413    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlg66\" (UniqueName: \"kubernetes.io/projected/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90-kube-api-access-rlg66\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:12.837456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.837427    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"
Nov 01 20:26:12.837641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.837452    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq" event={"ID":"bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90","Type":"ContainerDied","Data":"9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907"}
Nov 01 20:26:12.837641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:12.837500    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c879ffddfbe6e8741fb1d4737e6e1a41bcd379ab93caef198ed2a30080c5907"
Nov 01 20:26:17.021236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:17.021191    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"]
Nov 01 20:26:17.024886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:17.024865    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-361c20ca-9bhk9"]
Nov 01 20:26:18.431529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:18.431491    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9d99d9-6158-4cb6-841d-1e445d588bf6" path="/var/lib/kubelet/pods/bf9d99d9-6158-4cb6-841d-1e445d588bf6/volumes"
Nov 01 20:26:19.106592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.106558    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"]
Nov 01 20:26:19.107204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.107189    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" containerName="revoke-keys"
Nov 01 20:26:19.107261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.107205    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" containerName="revoke-keys"
Nov 01 20:26:19.107338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.107328    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" containerName="revoke-keys"
Nov 01 20:26:19.112060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.112043    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.118350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.118329    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"]
Nov 01 20:26:19.119294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.119270    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:26:19.119419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.119354    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:26:19.119419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.119378    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:26:19.119537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.119432    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:26:19.232672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.232637    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.232873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.232681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbnz\" (UniqueName: \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.233015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.232997    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.334626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.334594    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.334857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.334634    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbnz\" (UniqueName: \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.334908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.334874    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.335326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.335302    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.337212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.337193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.343133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.343111    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbnz\" (UniqueName: \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\") pod \"maas-api-revoke-sub-b7e8c515-52vbp\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.432967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.432870    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:19.756991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.756954    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"]
Nov 01 20:26:19.757956 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:26:19.757929    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211fed86_d69c_49d4_8c03_56b1cf7a8bcc.slice/crio-d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff WatchSource:0}: Error finding container d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff: Status 404 returned error can't find the container with id d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff
Nov 01 20:26:19.908944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:19.908912    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp" event={"ID":"211fed86-d69c-49d4-8c03-56b1cf7a8bcc","Type":"ContainerStarted","Data":"d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff"}
Nov 01 20:26:20.998810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:20.995805    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp" event={"ID":"211fed86-d69c-49d4-8c03-56b1cf7a8bcc","Type":"ContainerStarted","Data":"8def254543a77e028e6672b4c44ad54cf14eef1f2acf82f741f86a9fa50cc433"}
Nov 01 20:26:22.006268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.006226    5041 generic.go:356] "Generic (PLEG): container finished" podID="211fed86-d69c-49d4-8c03-56b1cf7a8bcc" containerID="8def254543a77e028e6672b4c44ad54cf14eef1f2acf82f741f86a9fa50cc433" exitCode=0
Nov 01 20:26:22.006707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.006296    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp" event={"ID":"211fed86-d69c-49d4-8c03-56b1cf7a8bcc","Type":"ContainerDied","Data":"8def254543a77e028e6672b4c44ad54cf14eef1f2acf82f741f86a9fa50cc433"}
Nov 01 20:26:22.189052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.189016    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:22.302045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.302005    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\") pod \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") "
Nov 01 20:26:22.302263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.302124    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\") pod \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") "
Nov 01 20:26:22.302263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.302215    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\" (UniqueName: \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\") pod \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\" (UID: \"211fed86-d69c-49d4-8c03-56b1cf7a8bcc\") "
Nov 01 20:26:22.302478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.302450    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca" pod "211fed86-d69c-49d4-8c03-56b1cf7a8bcc" (UID: "211fed86-d69c-49d4-8c03-56b1cf7a8bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:22.302873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.302849    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:22.304325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.304296    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca" pod "211fed86-d69c-49d4-8c03-56b1cf7a8bcc" (UID: "211fed86-d69c-49d4-8c03-56b1cf7a8bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:22.304449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.304349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz" pod "211fed86-d69c-49d4-8c03-56b1cf7a8bcc" (UID: "211fed86-d69c-49d4-8c03-56b1cf7a8bcc"). InnerVolumeSpecName "kube-api-access-qvbnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:22.404395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.404360    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:22.404395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:22.404390    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbnz\" (UniqueName: \"kubernetes.io/projected/211fed86-d69c-49d4-8c03-56b1cf7a8bcc-kube-api-access-qvbnz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:23.016470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:23.016428    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp" event={"ID":"211fed86-d69c-49d4-8c03-56b1cf7a8bcc","Type":"ContainerDied","Data":"d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff"}
Nov 01 20:26:23.016898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:23.016486    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3573e20db2e952109aaf3224b901fbfeb0fb4f6a6b0debdd612b296f9344aff"
Nov 01 20:26:23.016898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:23.016443    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"
Nov 01 20:26:28.542581 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:28.542547    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:26:30.021769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:30.021723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"]
Nov 01 20:26:30.025758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:30.025726    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91dd056-s5cqz"]
Nov 01 20:26:30.173275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:30.173247    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:26:30.430320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:30.430286    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc11cd7-5bd4-4a41-8db2-07e1c203ad91" path="/var/lib/kubelet/pods/0dc11cd7-5bd4-4a41-8db2-07e1c203ad91/volumes"
Nov 01 20:26:34.900337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:34.900294    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:26:35.852233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:35.852196    5041 scope.go:122] "RemoveContainer" containerID="6b3a9e32b0c5a3e560de8c3963561c028b38b2edac49f27426911308ca9ad344"
Nov 01 20:26:35.863225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:35.863195    5041 scope.go:122] "RemoveContainer" containerID="6b831f797b866706c32dfb7a6f06d0d1cf707c2ceb1d6846697409183898e37d"
Nov 01 20:26:35.873199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:35.873178    5041 scope.go:122] "RemoveContainer" containerID="8030bbd670adb085b9a168d13eaf243bfdc7e457da20e1a870202c7fe3d2680d"
Nov 01 20:26:35.882987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:35.882967    5041 scope.go:122] "RemoveContainer" containerID="c03ce3cabd592b1d68ceba28e7d2b28c4e8e7ee84ff0aa95af6460b591152301"
Nov 01 20:26:39.124222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:39.124191    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:26:39.124766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:39.124431    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy" containerID="cri-o://d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474" gracePeriod=30
Nov 01 20:26:40.382637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.382595    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"]
Nov 01 20:26:40.383233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.383218    5041 cpu_manager.go:414] "containerMap: removing container" podUID="211fed86-d69c-49d4-8c03-56b1cf7a8bcc" containerName="revoke-keys"
Nov 01 20:26:40.383280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.383233    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211fed86-d69c-49d4-8c03-56b1cf7a8bcc" containerName="revoke-keys"
Nov 01 20:26:40.383379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.383370    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="211fed86-d69c-49d4-8c03-56b1cf7a8bcc" containerName="revoke-keys"
Nov 01 20:26:40.388307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.388287    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.394269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.394241    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:26:40.394415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.394328    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:26:40.394415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.394358    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:26:40.394544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.394428    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:26:40.395632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.395603    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"]
Nov 01 20:26:40.430976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.430944    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq7g\" (UniqueName: \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.431174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.431004    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.431174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.431105    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.532797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.532735    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq7g\" (UniqueName: \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.533015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.532859    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.533015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.532977    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.534003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.533948    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.543521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.543492    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.545736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.545707    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq7g\" (UniqueName: \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\") pod \"maas-api-revoke-sub-e3bc02c6-6g6xj\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.712723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.712649    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:40.993171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:40.993107    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"]
Nov 01 20:26:40.993615 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:26:40.993575    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada3a6cb_0068_4557_87f2_77d1167fbf6f.slice/crio-0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4 WatchSource:0}: Error finding container 0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4: Status 404 returned error can't find the container with id 0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4
Nov 01 20:26:41.196551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:41.196513    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj" event={"ID":"ada3a6cb-0068-4557-87f2-77d1167fbf6f","Type":"ContainerStarted","Data":"0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4"}
Nov 01 20:26:42.210598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:42.210504    5041 generic.go:356] "Generic (PLEG): container finished" podID="ada3a6cb-0068-4557-87f2-77d1167fbf6f" containerID="ec47e0ad55513b4425d997520fcfdb9d462711c83d1b0b33465ffda795f94256" exitCode=0
Nov 01 20:26:42.210598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:42.210581    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj" event={"ID":"ada3a6cb-0068-4557-87f2-77d1167fbf6f","Type":"ContainerDied","Data":"ec47e0ad55513b4425d997520fcfdb9d462711c83d1b0b33465ffda795f94256"}
Nov 01 20:26:43.396545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.396516    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:43.487358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.487267    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\" (UniqueName: \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\") pod \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") "
Nov 01 20:26:43.487517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.487444    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\") pod \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") "
Nov 01 20:26:43.487586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.487556    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\") pod \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\" (UID: \"ada3a6cb-0068-4557-87f2-77d1167fbf6f\") "
Nov 01 20:26:43.488050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.488013    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca" pod "ada3a6cb-0068-4557-87f2-77d1167fbf6f" (UID: "ada3a6cb-0068-4557-87f2-77d1167fbf6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:43.489510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.489488    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g" pod "ada3a6cb-0068-4557-87f2-77d1167fbf6f" (UID: "ada3a6cb-0068-4557-87f2-77d1167fbf6f"). InnerVolumeSpecName "kube-api-access-xpq7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:43.489510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.489503    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca" pod "ada3a6cb-0068-4557-87f2-77d1167fbf6f" (UID: "ada3a6cb-0068-4557-87f2-77d1167fbf6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:43.589084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.589045    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpq7g\" (UniqueName: \"kubernetes.io/projected/ada3a6cb-0068-4557-87f2-77d1167fbf6f-kube-api-access-xpq7g\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:43.589084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.589077    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:43.589084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:43.589088    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3a6cb-0068-4557-87f2-77d1167fbf6f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.228403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.228375    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"
Nov 01 20:26:44.228584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.228401    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj" event={"ID":"ada3a6cb-0068-4557-87f2-77d1167fbf6f","Type":"ContainerDied","Data":"0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4"}
Nov 01 20:26:44.228584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.228459    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b191e9e5bff5da158dee12976521d1a49aab324c4faf56dc150ef61bd687bd4"
Nov 01 20:26:44.432851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.432818    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:26:44.500615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500585    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.500796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500642    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.500796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500741    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.500937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500813    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.500937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500880    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:44.501060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.500972    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:44.501204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.501185    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:44.505335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.505304    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:44.505567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.505539    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.505704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.505603    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.505704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.505646    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.505955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.505931    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:44.506321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.506299    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.506456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.506398    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\") pod \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\" (UID: \"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7\") "
Nov 01 20:26:44.507171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.507095    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.507171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.507125    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.507171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.507141    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.507171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.507154    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.507171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.507169    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.509159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.508771    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "kube-api-access-dstxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:44.514387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.514355    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:44.531087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.528582    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:26:44.531087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.529703    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy" pod "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" (UID: "b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:44.609088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.609056    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstxj\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-kube-api-access-dstxj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.609088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.609087    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.609290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.609097    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:44.609290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:44.609107    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.211827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.211751    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" podUID="39ae79db-b17d-4b0b-8756-316611154e4f" containerName="main" containerID="cri-o://f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0" gracePeriod=45
Nov 01 20:26:45.243003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.242969    5041 generic.go:356] "Generic (PLEG): container finished" podID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerID="d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474" exitCode=0
Nov 01 20:26:45.243136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.243021    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"
Nov 01 20:26:45.243136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.243042    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerDied","Data":"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"}
Nov 01 20:26:45.243136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.243099    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb" event={"ID":"b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7","Type":"ContainerDied","Data":"56d0cdb2306362815d279793212ac3ab83432674ed39b87e8a3f99d03a60e2f5"}
Nov 01 20:26:45.243136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.243121    5041 scope.go:122] "RemoveContainer" containerID="d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"
Nov 01 20:26:45.313970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.313941    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:26:45.318874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.318834    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2babaef6-openshift-default-6858cf4d9c-4gcxb"]
Nov 01 20:26:45.326936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.326762    5041 scope.go:122] "RemoveContainer" containerID="2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"
Nov 01 20:26:45.337457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.337438    5041 scope.go:122] "RemoveContainer" containerID="d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"
Nov 01 20:26:45.337735 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:45.337714    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474\": container with ID starting with d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474 not found: ID does not exist" containerID="d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"
Nov 01 20:26:45.337927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.337748    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474"} err="failed to get container status \"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474\": rpc error: code = NotFound desc = could not find container \"d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474\": container with ID starting with d87ace68025249ab8ef238a14e705bbfa8ab5463e67d18a0a0e1b716e7696474 not found: ID does not exist"
Nov 01 20:26:45.337927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.337768    5041 scope.go:122] "RemoveContainer" containerID="2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"
Nov 01 20:26:45.338067 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:45.337987    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82\": container with ID starting with 2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82 not found: ID does not exist" containerID="2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"
Nov 01 20:26:45.338067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.338008    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82"} err="failed to get container status \"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82\": rpc error: code = NotFound desc = could not find container \"2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82\": container with ID starting with 2b7b8f49310fd0673ddd900fd55262b6c018f41e43118b412a74f6964c34fc82 not found: ID does not exist"
Nov 01 20:26:45.513933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.513906    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:26:45.622674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622635    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.622916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622686    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\" (UniqueName: \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.622916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622738    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.622916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622831    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.622916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622903    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.623137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.622948    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\") pod \"39ae79db-b17d-4b0b-8756-316611154e4f\" (UID: \"39ae79db-b17d-4b0b-8756-316611154e4f\") "
Nov 01 20:26:45.623137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623006    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:45.623137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623093    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:45.623320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623166    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:45.623479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623456    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.623479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623478    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.623641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.623492    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.625064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.625034    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "kube-api-access-m8jsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:45.625177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.625062    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:45.625177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.625112    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm" pod "39ae79db-b17d-4b0b-8756-316611154e4f" (UID: "39ae79db-b17d-4b0b-8756-316611154e4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:45.724824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.724762    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8jsz\" (UniqueName: \"kubernetes.io/projected/39ae79db-b17d-4b0b-8756-316611154e4f-kube-api-access-m8jsz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.724824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.724819    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ae79db-b17d-4b0b-8756-316611154e4f-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:45.724824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:45.724829    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ae79db-b17d-4b0b-8756-316611154e4f-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:46.253264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.253230    5041 generic.go:356] "Generic (PLEG): container finished" podID="39ae79db-b17d-4b0b-8756-316611154e4f" containerID="f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0" exitCode=0
Nov 01 20:26:46.253458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.253271    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"
Nov 01 20:26:46.253458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.253321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" event={"ID":"39ae79db-b17d-4b0b-8756-316611154e4f","Type":"ContainerDied","Data":"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"}
Nov 01 20:26:46.253458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.253376    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl" event={"ID":"39ae79db-b17d-4b0b-8756-316611154e4f","Type":"ContainerDied","Data":"0236bb6f451456e22e1929b2e0bcf06511e1a325acb55b07c4d33b3859162b79"}
Nov 01 20:26:46.253458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.253401    5041 scope.go:122] "RemoveContainer" containerID="f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"
Nov 01 20:26:46.265048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.265029    5041 scope.go:122] "RemoveContainer" containerID="f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"
Nov 01 20:26:46.265318 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:46.265298    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0\": container with ID starting with f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0 not found: ID does not exist" containerID="f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"
Nov 01 20:26:46.265400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.265325    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0"} err="failed to get container status \"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0\": rpc error: code = NotFound desc = could not find container \"f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0\": container with ID starting with f1d784301ee76e18c04d66a936d0aa1a00b250755683861bf85fd6c6ffcf48e0 not found: ID does not exist"
Nov 01 20:26:46.277373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.277347    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:26:46.281547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.281524    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/rate-test-model-8b6cd6-kserve-559f954b85-7gzkl"]
Nov 01 20:26:46.452050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.448902    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ae79db-b17d-4b0b-8756-316611154e4f" path="/var/lib/kubelet/pods/39ae79db-b17d-4b0b-8756-316611154e4f/volumes"
Nov 01 20:26:46.452050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:46.449810    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" path="/var/lib/kubelet/pods/b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7/volumes"
Nov 01 20:26:47.855219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855187    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"]
Nov 01 20:26:47.855817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855802    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.855874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855818    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.855874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855832    5041 cpu_manager.go:414] "containerMap: removing container" podUID="39ae79db-b17d-4b0b-8756-316611154e4f" containerName="main"
Nov 01 20:26:47.855874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855839    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ae79db-b17d-4b0b-8756-316611154e4f" containerName="main"
Nov 01 20:26:47.855874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855852    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.855874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855857    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855880    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ada3a6cb-0068-4557-87f2-77d1167fbf6f" containerName="revoke-keys"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855885    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada3a6cb-0068-4557-87f2-77d1167fbf6f" containerName="revoke-keys"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855975    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855984    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ae79db-b17d-4b0b-8756-316611154e4f" containerName="main"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.855991    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90c22fc-b467-4b0c-a7b7-20b57ac9b6d7" containerName="istio-proxy"
Nov 01 20:26:47.856041 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.856001    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada3a6cb-0068-4557-87f2-77d1167fbf6f" containerName="revoke-keys"
Nov 01 20:26:47.861068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.861046    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:47.864658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.864632    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:26:47.864821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.864739    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:26:47.864965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.864948    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:26:47.865044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.865034    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:26:47.866445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.866423    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"]
Nov 01 20:26:47.955011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.954980    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:47.955171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.955124    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:47.955291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:47.955278    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7h6c\" (UniqueName: \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.056309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.056270    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.056498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.056482    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7h6c\" (UniqueName: \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.056612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.056595    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.057030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.057009    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.059099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.059069    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.064246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.064216    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7h6c\" (UniqueName: \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\") pod \"maas-api-revoke-sub-c7888f16-p2tvw\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.178692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.178591    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:48.463302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:48.463271    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"]
Nov 01 20:26:48.463638 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:26:48.463616    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1472eaab_f207_4de5_b138_86210a4ce907.slice/crio-9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f WatchSource:0}: Error finding container 9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f: Status 404 returned error can't find the container with id 9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f
Nov 01 20:26:49.284240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:49.284140    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw" event={"ID":"1472eaab-f207-4de5-b138-86210a4ce907","Type":"ContainerStarted","Data":"128a30801fc06eccdf609d02e1135e11d911654877bce159d1a178d2b5c9ce11"}
Nov 01 20:26:49.284240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:49.284210    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw" event={"ID":"1472eaab-f207-4de5-b138-86210a4ce907","Type":"ContainerStarted","Data":"9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f"}
Nov 01 20:26:49.301545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:49.301491    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw" podStartSLOduration=2.3014780950000002 podStartE2EDuration="2.301478095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:49.298020438 +0000 UTC m=+1697.019099116" watchObservedRunningTime="2026-11-01 20:26:49.301478095 +0000 UTC m=+1697.022556770"
Nov 01 20:26:49.932195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:49.932159    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerName="main" containerID="cri-o://34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1" gracePeriod=45
Nov 01 20:26:50.251164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.251139    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:26:50.296383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.296342    5041 generic.go:356] "Generic (PLEG): container finished" podID="1472eaab-f207-4de5-b138-86210a4ce907" containerID="128a30801fc06eccdf609d02e1135e11d911654877bce159d1a178d2b5c9ce11" exitCode=0
Nov 01 20:26:50.296910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.296421    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw" event={"ID":"1472eaab-f207-4de5-b138-86210a4ce907","Type":"ContainerDied","Data":"128a30801fc06eccdf609d02e1135e11d911654877bce159d1a178d2b5c9ce11"}
Nov 01 20:26:50.297980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.297955    5041 generic.go:356] "Generic (PLEG): container finished" podID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerID="34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1" exitCode=0
Nov 01 20:26:50.298098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.298000    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"
Nov 01 20:26:50.298098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.298032    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" event={"ID":"3ee0f5ac-0817-4ee1-af87-afcebd2d9982","Type":"ContainerDied","Data":"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"}
Nov 01 20:26:50.298098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.298068    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt" event={"ID":"3ee0f5ac-0817-4ee1-af87-afcebd2d9982","Type":"ContainerDied","Data":"12047c3e83c08003193b5db38222387ade29261bf6210c92cd707f1d8b4abb0a"}
Nov 01 20:26:50.298098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.298087    5041 scope.go:122] "RemoveContainer" containerID="34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"
Nov 01 20:26:50.319332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.319309    5041 scope.go:122] "RemoveContainer" containerID="34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"
Nov 01 20:26:50.319710 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:50.319676    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1\": container with ID starting with 34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1 not found: ID does not exist" containerID="34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"
Nov 01 20:26:50.319852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.319721    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1"} err="failed to get container status \"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1\": rpc error: code = NotFound desc = could not find container \"34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1\": container with ID starting with 34eabc40d5116d9aa50e1b03d684408ed29990d8dd4bed1f6d6ef2e76656bdf1 not found: ID does not exist"
Nov 01 20:26:50.392628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.392804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392748    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.392855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392818    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.392855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392846    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.392937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392925    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\" (UniqueName: \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.393001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.392981    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\") pod \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\" (UID: \"3ee0f5ac-0817-4ee1-af87-afcebd2d9982\") "
Nov 01 20:26:50.393061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393032    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:50.393155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393136    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:50.393319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:50.393633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393612    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.393689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393640    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.393689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.393656    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.394996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.394972    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:26:50.395099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.395057    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:50.395166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.395138    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54" pod "3ee0f5ac-0817-4ee1-af87-afcebd2d9982" (UID: "3ee0f5ac-0817-4ee1-af87-afcebd2d9982"). InnerVolumeSpecName "kube-api-access-lvq54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:50.495299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.495224    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.495299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.495257    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.495299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.495269    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvq54\" (UniqueName: \"kubernetes.io/projected/3ee0f5ac-0817-4ee1-af87-afcebd2d9982-kube-api-access-lvq54\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:50.624398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.624359    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:26:50.628666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:50.628638    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/rate-test-model-036c69-kserve-584595bf69-r99wt"]
Nov 01 20:26:51.481158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.481128    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:51.610302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.610210    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\") pod \"1472eaab-f207-4de5-b138-86210a4ce907\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") "
Nov 01 20:26:51.610479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.610323    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\") pod \"1472eaab-f207-4de5-b138-86210a4ce907\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") "
Nov 01 20:26:51.610479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.610447    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\" (UniqueName: \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\") pod \"1472eaab-f207-4de5-b138-86210a4ce907\" (UID: \"1472eaab-f207-4de5-b138-86210a4ce907\") "
Nov 01 20:26:51.610794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.610732    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca" pod "1472eaab-f207-4de5-b138-86210a4ce907" (UID: "1472eaab-f207-4de5-b138-86210a4ce907"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:51.611008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.610983    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:51.612373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.612351    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca" pod "1472eaab-f207-4de5-b138-86210a4ce907" (UID: "1472eaab-f207-4de5-b138-86210a4ce907"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:51.612545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.612521    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c" pod "1472eaab-f207-4de5-b138-86210a4ce907" (UID: "1472eaab-f207-4de5-b138-86210a4ce907"). InnerVolumeSpecName "kube-api-access-x7h6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:51.712766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.712725    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1472eaab-f207-4de5-b138-86210a4ce907-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:51.712766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:51.712760    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7h6c\" (UniqueName: \"kubernetes.io/projected/1472eaab-f207-4de5-b138-86210a4ce907-kube-api-access-x7h6c\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:52.317854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:52.317818    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"
Nov 01 20:26:52.318026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:52.317829    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw" event={"ID":"1472eaab-f207-4de5-b138-86210a4ce907","Type":"ContainerDied","Data":"9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f"}
Nov 01 20:26:52.318026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:52.317989    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9699bd8c68e6236e9f498f1999f83096eeda528546b70766aa259247c1042c1f"
Nov 01 20:26:52.431255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:52.431221    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" path="/var/lib/kubelet/pods/3ee0f5ac-0817-4ee1-af87-afcebd2d9982/volumes"
Nov 01 20:26:53.173801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.172824    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"]
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.173859    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1472eaab-f207-4de5-b138-86210a4ce907" containerName="revoke-keys"
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.173882    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1472eaab-f207-4de5-b138-86210a4ce907" containerName="revoke-keys"
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.173906    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerName="main"
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.173916    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerName="main"
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.174092    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1472eaab-f207-4de5-b138-86210a4ce907" containerName="revoke-keys"
Nov 01 20:26:53.174298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.174108    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee0f5ac-0817-4ee1-af87-afcebd2d9982" containerName="main"
Nov 01 20:26:53.197800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.188431    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.197800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.194729    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:26:53.214384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.212263    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:26:53.214384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.212354    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:26:53.214384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.212429    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:26:53.269909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.256623    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.269909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.256716    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.269909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.256800    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.364885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.364848    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.365531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.365509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.365979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.365958    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.384465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.384426    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"]
Nov 01 20:26:53.393023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.392940    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.419540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.419468    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.433740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.433663    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee514429-c6thj\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:53.709992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:53.709897    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:54.038391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:54.038353    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"]
Nov 01 20:26:54.722265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:54.722224    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj" event={"ID":"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca","Type":"ContainerStarted","Data":"0bce00f362c0631990251c49326bfd8be108b17d197142a0fac38274f0cb59e3"}
Nov 01 20:26:55.846857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:55.846818    5041 generic.go:356] "Generic (PLEG): container finished" podID="c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" containerID="16e41fe967ebd19989140f35bec8e6f2d52d64053ceeb9ad32e5bc65b6d7fb1d" exitCode=0
Nov 01 20:26:55.847633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:55.847602    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj" event={"ID":"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca","Type":"ContainerDied","Data":"16e41fe967ebd19989140f35bec8e6f2d52d64053ceeb9ad32e5bc65b6d7fb1d"}
Nov 01 20:26:55.928901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:55.923168    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": context deadline exceeded"
Nov 01 20:26:57.045737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.045701    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:57.087452 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.087413    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\") pod \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") "
Nov 01 20:26:57.087654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.087620    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\") pod \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") "
Nov 01 20:26:57.087828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.087807    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\") pod \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\" (UID: \"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca\") "
Nov 01 20:26:57.087928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.087879    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca" pod "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" (UID: "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:26:57.088521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.088377    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:57.090968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.090926    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca" pod "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" (UID: "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:26:57.091521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.091493    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh" pod "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" (UID: "c5cae964-2241-42c0-a74f-dc4b9ff3e4ca"). InnerVolumeSpecName "kube-api-access-9trxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:26:57.190232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.190063    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:57.190232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.190103    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca-kube-api-access-9trxh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:26:57.868573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.868543    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"
Nov 01 20:26:57.868750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.868574    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj" event={"ID":"c5cae964-2241-42c0-a74f-dc4b9ff3e4ca","Type":"ContainerDied","Data":"0bce00f362c0631990251c49326bfd8be108b17d197142a0fac38274f0cb59e3"}
Nov 01 20:26:57.868750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.868629    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bce00f362c0631990251c49326bfd8be108b17d197142a0fac38274f0cb59e3"
Nov 01 20:26:57.966210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.966170    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:26:57.967185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.967160    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" containerName="revoke-keys"
Nov 01 20:26:57.967185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.967185    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" containerName="revoke-keys"
Nov 01 20:26:57.967462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.967413    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" containerName="revoke-keys"
Nov 01 20:26:57.971725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.971702    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:57.979422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.979397    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"openshift-service-ca.crt\""
Nov 01 20:26:57.979717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.979698    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"sub-test-model-036c69-kserve-self-signed-certs\""
Nov 01 20:26:57.979974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.979956    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"kube-root-ca.crt\""
Nov 01 20:26:57.980124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.979969    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:26:57.980198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:57.980080    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-036c69\"/\"default-dockercfg-w9jbz\""
Nov 01 20:26:58.007112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007078    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.007260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007146    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.007308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007261    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.007345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007325    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.007400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007384    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.007628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.007609    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmlq\" (UniqueName: \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108384    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmlq\" (UniqueName: \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108428    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108473    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108509    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108558    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.108955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.108610    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.109175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.109003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.109308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.109257    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.109308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.109276    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.111023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.110996    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.111204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.111187    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.141593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.140509    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmlq\" (UniqueName: \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\") pod \"sub-test-model-036c69-kserve-66cbb89ff-j6fxk\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") " pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.293047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.293009    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:26:58.547244 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:26:58.547206    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:26:58.595095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.595069    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:26:58.885550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.885456    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" event={"ID":"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad","Type":"ContainerStarted","Data":"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"}
Nov 01 20:26:58.885550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.885503    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" event={"ID":"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad","Type":"ContainerStarted","Data":"b4497dd41dc11acf660c7ed31380b3ee0d660c22c3d380dd36cf4b060e414479"}
Nov 01 20:26:58.908668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:58.908611    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" podStartSLOduration=1.9085973410000001 podStartE2EDuration="1.908597341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:58.905377919 +0000 UTC m=+1706.626456597" watchObservedRunningTime="2026-11-01 20:26:58.908597341 +0000 UTC m=+1706.629676016"
Nov 01 20:26:59.293792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:26:59.293745    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:27:08.298349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:08.295305    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:27:08.323791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:08.323739    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:27:09.597307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:09.597274    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:27:28.465687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.465654    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"]
Nov 01 20:27:28.470148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.470118    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"]
Nov 01 20:27:28.470297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.470233    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.473893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.473865    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:27:28.474992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.474971    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:27:28.475117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.475082    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:27:28.475225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.475208    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:27:28.542286 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:27:28.542249    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:27:28.557322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.557292    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.557512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.557492    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96pqv\" (UniqueName: \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.557634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.557614    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.658883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.658844    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.659061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.658950    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.659106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.659062    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96pqv\" (UniqueName: \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.659517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.659493    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.661398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.661373    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.666279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.666250    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96pqv\" (UniqueName: \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\") pod \"maas-api-revoke-sub-e6f85707-mhwjh\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:28.801837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:28.801766    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:29.084378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:29.084349    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"]
Nov 01 20:27:29.085464 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:27:29.085438    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cfc601_57ab_4780_83c3_65e20b7bf0dd.slice/crio-d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed WatchSource:0}: Error finding container d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed: Status 404 returned error can't find the container with id d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed
Nov 01 20:27:29.283567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:29.283533    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh" event={"ID":"a7cfc601-57ab-4780-83c3-65e20b7bf0dd","Type":"ContainerStarted","Data":"d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed"}
Nov 01 20:27:30.292702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:30.292668    5041 generic.go:356] "Generic (PLEG): container finished" podID="a7cfc601-57ab-4780-83c3-65e20b7bf0dd" containerID="fc8eb43d1fc851a3488d521692879553dbba6b4def1c18032e2ac94ebf2e67d4" exitCode=0
Nov 01 20:27:30.293178 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:30.292761    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh" event={"ID":"a7cfc601-57ab-4780-83c3-65e20b7bf0dd","Type":"ContainerDied","Data":"fc8eb43d1fc851a3488d521692879553dbba6b4def1c18032e2ac94ebf2e67d4"}
Nov 01 20:27:31.472087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.472060    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:31.598083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.597997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\") pod \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") "
Nov 01 20:27:31.598226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.598078    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\") pod \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") "
Nov 01 20:27:31.598268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.598223    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\" (UniqueName: \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\") pod \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\" (UID: \"a7cfc601-57ab-4780-83c3-65e20b7bf0dd\") "
Nov 01 20:27:31.598461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.598433    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca" pod "a7cfc601-57ab-4780-83c3-65e20b7bf0dd" (UID: "a7cfc601-57ab-4780-83c3-65e20b7bf0dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:27:31.598915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.598886    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:31.600394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.600366    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca" pod "a7cfc601-57ab-4780-83c3-65e20b7bf0dd" (UID: "a7cfc601-57ab-4780-83c3-65e20b7bf0dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:27:31.600498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.600441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv" pod "a7cfc601-57ab-4780-83c3-65e20b7bf0dd" (UID: "a7cfc601-57ab-4780-83c3-65e20b7bf0dd"). InnerVolumeSpecName "kube-api-access-96pqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:27:31.700680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.700646    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96pqv\" (UniqueName: \"kubernetes.io/projected/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-kube-api-access-96pqv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:31.700680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:31.700675    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cfc601-57ab-4780-83c3-65e20b7bf0dd-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:32.309136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:32.309108    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"
Nov 01 20:27:32.309136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:32.309130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh" event={"ID":"a7cfc601-57ab-4780-83c3-65e20b7bf0dd","Type":"ContainerDied","Data":"d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed"}
Nov 01 20:27:32.309360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:32.309172    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d613a9de720bd0ce93e5c5bc717b67f773c7bb06111b38ea2808b86b9f7785ed"
Nov 01 20:27:35.942176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:35.942143    5041 scope.go:122] "RemoveContainer" containerID="cb0400f7e801eec571560f56a3640d66bf34b4474be3c77f6392e7be98a414b7"
Nov 01 20:27:35.953946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:35.953924    5041 scope.go:122] "RemoveContainer" containerID="c5993555db9898b077f8ce7f4fa694537112a0784a1b4fa4b84a3330cf78cdd1"
Nov 01 20:27:35.964205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:35.964182    5041 scope.go:122] "RemoveContainer" containerID="9bc7632f3ef6dba346b5d0d219935c945977b4ae1972065656ef4e8ce31c277c"
Nov 01 20:27:35.974701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:35.974678    5041 scope.go:122] "RemoveContainer" containerID="0f5598195250f91a04064848ef5803c660f1cec1096d9367f92d8968ee30d107"
Nov 01 20:27:35.984199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:35.984175    5041 scope.go:122] "RemoveContainer" containerID="8ac9d6f3bdd745d944986c1a8233a4a051afd7d55166d17a01c164edd2526682"
Nov 01 20:27:39.816614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.816581    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"]
Nov 01 20:27:39.817541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.817519    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a7cfc601-57ab-4780-83c3-65e20b7bf0dd" containerName="revoke-keys"
Nov 01 20:27:39.817590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.817546    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cfc601-57ab-4780-83c3-65e20b7bf0dd" containerName="revoke-keys"
Nov 01 20:27:39.817767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.817754    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cfc601-57ab-4780-83c3-65e20b7bf0dd" containerName="revoke-keys"
Nov 01 20:27:39.823585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.823557    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.827607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.827557    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"]
Nov 01 20:27:39.827978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.827956    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:27:39.828988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.828971    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:27:39.848379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.848346    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.848501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.848480    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rfbw\" (UniqueName: \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.848595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.848584    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.890537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.890507    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:27:39.890537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.890507    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:27:39.949952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.949914    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.950118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.950029    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.950168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.950131    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rfbw\" (UniqueName: \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.950861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.950824    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.952503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.952479    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:39.957340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:39.957300    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rfbw\" (UniqueName: \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\") pod \"maas-api-revoke-sub-4419b4ae-zp74v\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:40.143719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:40.143634    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:40.433979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:40.433948    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"]
Nov 01 20:27:40.434385 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:27:40.434360    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa3115e_595e_4468_84d0_adfe2fc0822f.slice/crio-aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e WatchSource:0}: Error finding container aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e: Status 404 returned error can't find the container with id aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e
Nov 01 20:27:41.387935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:41.387898    5041 generic.go:356] "Generic (PLEG): container finished" podID="efa3115e-595e-4468-84d0-adfe2fc0822f" containerID="dee6de1f74e738d7ad18f938c82a029f183ecd98c41a5f3cc5a860b685ebd9b5" exitCode=0
Nov 01 20:27:41.388363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:41.387982    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v" event={"ID":"efa3115e-595e-4468-84d0-adfe2fc0822f","Type":"ContainerDied","Data":"dee6de1f74e738d7ad18f938c82a029f183ecd98c41a5f3cc5a860b685ebd9b5"}
Nov 01 20:27:41.388363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:41.388069    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v" event={"ID":"efa3115e-595e-4468-84d0-adfe2fc0822f","Type":"ContainerStarted","Data":"aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e"}
Nov 01 20:27:42.866432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.866304    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:42.919035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.918996    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\") pod \"efa3115e-595e-4468-84d0-adfe2fc0822f\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") "
Nov 01 20:27:42.919309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.919277    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\") pod \"efa3115e-595e-4468-84d0-adfe2fc0822f\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") "
Nov 01 20:27:42.919470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.919393    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\" (UniqueName: \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\") pod \"efa3115e-595e-4468-84d0-adfe2fc0822f\" (UID: \"efa3115e-595e-4468-84d0-adfe2fc0822f\") "
Nov 01 20:27:42.921021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.920986    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca" pod "efa3115e-595e-4468-84d0-adfe2fc0822f" (UID: "efa3115e-595e-4468-84d0-adfe2fc0822f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:27:42.923171 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.923146    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw" pod "efa3115e-595e-4468-84d0-adfe2fc0822f" (UID: "efa3115e-595e-4468-84d0-adfe2fc0822f"). InnerVolumeSpecName "kube-api-access-6rfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:27:42.925835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:42.925769    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca" pod "efa3115e-595e-4468-84d0-adfe2fc0822f" (UID: "efa3115e-595e-4468-84d0-adfe2fc0822f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:27:43.021762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.021726    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:43.021762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.021758    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa3115e-595e-4468-84d0-adfe2fc0822f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:43.021762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.021770    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rfbw\" (UniqueName: \"kubernetes.io/projected/efa3115e-595e-4468-84d0-adfe2fc0822f-kube-api-access-6rfbw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:27:43.418409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.418361    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"
Nov 01 20:27:43.418584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.418368    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v" event={"ID":"efa3115e-595e-4468-84d0-adfe2fc0822f","Type":"ContainerDied","Data":"aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e"}
Nov 01 20:27:43.418584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:27:43.418519    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaedbda09ddb73f99aedc71c070927cc7feb5769b09308e2c64ee3eb39d32e0e"
Nov 01 20:27:58.542846 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:27:58.542805    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:28:08.022880 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:08.022827    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"]
Nov 01 20:28:08.026503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:08.026479    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793758de-tbkmp"]
Nov 01 20:28:08.429607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:08.429565    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a66e20-aecc-435d-8d94-e07281b56717" path="/var/lib/kubelet/pods/62a66e20-aecc-435d-8d94-e07281b56717/volumes"
Nov 01 20:28:28.562720 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:28:28.562679    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:28:29.053623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.053586    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:28:29.054573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.054549    5041 cpu_manager.go:414] "containerMap: removing container" podUID="efa3115e-595e-4468-84d0-adfe2fc0822f" containerName="revoke-keys"
Nov 01 20:28:29.054657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.054576    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa3115e-595e-4468-84d0-adfe2fc0822f" containerName="revoke-keys"
Nov 01 20:28:29.054850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.054831    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa3115e-595e-4468-84d0-adfe2fc0822f" containerName="revoke-keys"
Nov 01 20:28:29.060012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.059989    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.065475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.065443    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d2dc0ad5-openshift-default-dockercfg-kcnmb\""
Nov 01 20:28:29.066042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.066014    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:28:29.190586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190543    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.190835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190669    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.190835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190715    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.190835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190754    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.191038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190883    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.191038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190926    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.191038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.190956    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.191204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.191070    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.191204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.191172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svn58\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.304862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304167    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.304862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304338    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.304862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304505    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.304862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304559    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.304862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304719    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.305304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.304983    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svn58\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.305437    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.305701    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.306423    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.307738    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.308162    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.308772    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.309689    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.327803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.320236    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.368085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.368015    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.376342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.375315    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.376342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.375688    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svn58\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.376958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.376932    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\") pod \"e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") " pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.390861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.388054    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:29.803305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.803207    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:28:29.803858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.803824    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:28:29.804009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.803994    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:28:29.823288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.823258    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:28:29.872975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:29.872943    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" event={"ID":"a4d0f35a-b93d-4245-8745-7f93722536cd","Type":"ContainerStarted","Data":"f1d8284f6fb0a83fb9fdcc36ce808f920b10ab10abd9e533f3e4893a2dc17ef7"}
Nov 01 20:28:30.882427 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:30.882386    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" event={"ID":"a4d0f35a-b93d-4245-8745-7f93722536cd","Type":"ContainerStarted","Data":"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"}
Nov 01 20:28:30.905086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:30.905037    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" podStartSLOduration=1.905023438 podStartE2EDuration="1.905023438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:30.903379682 +0000 UTC m=+1798.624458369" watchObservedRunningTime="2026-11-01 20:28:30.905023438 +0000 UTC m=+1798.626102114"
Nov 01 20:28:31.388503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:31.388467    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:31.394016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:31.393991    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:31.890614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:31.890560    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:31.891483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:31.891462    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:28:32.884905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:32.884871    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:28:32.886382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:32.886359    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:28:33.425492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.425452    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"]
Nov 01 20:28:33.430221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.430149    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.434038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.434008    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:28:33.434199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.434182    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:28:33.435247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.435226    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:28:33.435370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.435305    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:28:33.435436 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.435375    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"]
Nov 01 20:28:33.485448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.485410    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.485619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.485599    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.485909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.485884    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8k9\" (UniqueName: \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.587275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.587238    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8k9\" (UniqueName: \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.587457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.587320    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.587457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.587434    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.588052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.588028    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.589827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.589808    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.595255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.595230    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8k9\" (UniqueName: \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\") pod \"maas-api-revoke-sub-a8a0f5b9-tvbsv\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.748761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.748665    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:33.983744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.983710    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"]
Nov 01 20:28:33.988902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.988881    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:33.997147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:33.997119    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"]
Nov 01 20:28:34.032409 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:28:34.032381    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb933f6_4d42_4351_89e1_f7ba574c74f8.slice/crio-00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465 WatchSource:0}: Error finding container 00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465: Status 404 returned error can't find the container with id 00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465
Nov 01 20:28:34.033160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.033136    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"]
Nov 01 20:28:34.094252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.094219    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.094378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.094332    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcxbz\" (UniqueName: \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.094378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.094374    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.195373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.195337    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.195567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.195405    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcxbz\" (UniqueName: \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.195567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.195429    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.196308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.196272    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.198206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.198178    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.213531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.213497    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcxbz\" (UniqueName: \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\") pod \"maas-api-revoke-sub-d1ea27b9-phc5p\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.307700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.307666    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:34.724382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.724347    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"]
Nov 01 20:28:34.933993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.933951    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p" event={"ID":"bc4be621-8e5f-4a64-ba35-c73af038444c","Type":"ContainerStarted","Data":"40f7b6efbfbe50cea4f9383e7d620ec08680c2f97e2b7102e61ad207fa760a57"}
Nov 01 20:28:34.935547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.935519    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv" event={"ID":"ffb933f6-4d42-4351-89e1-f7ba574c74f8","Type":"ContainerStarted","Data":"d8997fdba9be15c5addf9b986bc5e99255dea7fa712e0fcfa835391dad6c8fde"}
Nov 01 20:28:34.935671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.935566    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv" event={"ID":"ffb933f6-4d42-4351-89e1-f7ba574c74f8","Type":"ContainerStarted","Data":"00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465"}
Nov 01 20:28:34.955739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:34.955562    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv" podStartSLOduration=1.95554811 podStartE2EDuration="1.95554811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:34.950360595 +0000 UTC m=+1802.671439271" watchObservedRunningTime="2026-11-01 20:28:34.95554811 +0000 UTC m=+1802.676626786"
Nov 01 20:28:35.945292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:35.945258    5041 generic.go:356] "Generic (PLEG): container finished" podID="bc4be621-8e5f-4a64-ba35-c73af038444c" containerID="8d02b186e62a3ce6e82a563161f27c6c66bd139a21ec67f3dcd324ed27503fa8" exitCode=0
Nov 01 20:28:35.945731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:35.945346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p" event={"ID":"bc4be621-8e5f-4a64-ba35-c73af038444c","Type":"ContainerDied","Data":"8d02b186e62a3ce6e82a563161f27c6c66bd139a21ec67f3dcd324ed27503fa8"}
Nov 01 20:28:35.947268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:35.947246    5041 generic.go:356] "Generic (PLEG): container finished" podID="ffb933f6-4d42-4351-89e1-f7ba574c74f8" containerID="d8997fdba9be15c5addf9b986bc5e99255dea7fa712e0fcfa835391dad6c8fde" exitCode=0
Nov 01 20:28:35.947414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:35.947333    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv" event={"ID":"ffb933f6-4d42-4351-89e1-f7ba574c74f8","Type":"ContainerDied","Data":"d8997fdba9be15c5addf9b986bc5e99255dea7fa712e0fcfa835391dad6c8fde"}
Nov 01 20:28:37.132522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.132494    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:37.162102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.162065    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\") pod \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") "
Nov 01 20:28:37.162307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.162159    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\") pod \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") "
Nov 01 20:28:37.162307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.162270    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\" (UniqueName: \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\") pod \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\" (UID: \"ffb933f6-4d42-4351-89e1-f7ba574c74f8\") "
Nov 01 20:28:37.163212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.163179    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca" pod "ffb933f6-4d42-4351-89e1-f7ba574c74f8" (UID: "ffb933f6-4d42-4351-89e1-f7ba574c74f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:37.164834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.164808    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca" pod "ffb933f6-4d42-4351-89e1-f7ba574c74f8" (UID: "ffb933f6-4d42-4351-89e1-f7ba574c74f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:37.165365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.165337    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9" pod "ffb933f6-4d42-4351-89e1-f7ba574c74f8" (UID: "ffb933f6-4d42-4351-89e1-f7ba574c74f8"). InnerVolumeSpecName "kube-api-access-ql8k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:37.198705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.198676    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:37.263816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.263693    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\") pod \"bc4be621-8e5f-4a64-ba35-c73af038444c\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") "
Nov 01 20:28:37.263816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.263754    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\") pod \"bc4be621-8e5f-4a64-ba35-c73af038444c\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") "
Nov 01 20:28:37.264023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.263997    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\" (UniqueName: \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\") pod \"bc4be621-8e5f-4a64-ba35-c73af038444c\" (UID: \"bc4be621-8e5f-4a64-ba35-c73af038444c\") "
Nov 01 20:28:37.264257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.264229    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca" pod "bc4be621-8e5f-4a64-ba35-c73af038444c" (UID: "bc4be621-8e5f-4a64-ba35-c73af038444c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:37.264657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.264619    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.264657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.264650    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.264864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.264740    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8k9\" (UniqueName: \"kubernetes.io/projected/ffb933f6-4d42-4351-89e1-f7ba574c74f8-kube-api-access-ql8k9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.264864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.264756    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb933f6-4d42-4351-89e1-f7ba574c74f8-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.266118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.266094    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca" pod "bc4be621-8e5f-4a64-ba35-c73af038444c" (UID: "bc4be621-8e5f-4a64-ba35-c73af038444c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:37.266595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.266576    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz" pod "bc4be621-8e5f-4a64-ba35-c73af038444c" (UID: "bc4be621-8e5f-4a64-ba35-c73af038444c"). InnerVolumeSpecName "kube-api-access-qcxbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:37.366666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.366621    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc4be621-8e5f-4a64-ba35-c73af038444c-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.366666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.366667    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcxbz\" (UniqueName: \"kubernetes.io/projected/bc4be621-8e5f-4a64-ba35-c73af038444c-kube-api-access-qcxbz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:37.964000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.963967    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"
Nov 01 20:28:37.964202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.964002    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p" event={"ID":"bc4be621-8e5f-4a64-ba35-c73af038444c","Type":"ContainerDied","Data":"40f7b6efbfbe50cea4f9383e7d620ec08680c2f97e2b7102e61ad207fa760a57"}
Nov 01 20:28:37.964202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.964050    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f7b6efbfbe50cea4f9383e7d620ec08680c2f97e2b7102e61ad207fa760a57"
Nov 01 20:28:37.965476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.965445    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv" event={"ID":"ffb933f6-4d42-4351-89e1-f7ba574c74f8","Type":"ContainerDied","Data":"00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465"}
Nov 01 20:28:37.965610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.965494    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00e980471d55decc5269cb8382a3bd827b23d4234525142f192736bd05c2f465"
Nov 01 20:28:37.965610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:37.965455    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"
Nov 01 20:28:38.005604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:38.005576    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:28:39.021903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:39.021870    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"]
Nov 01 20:28:39.025812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:39.025774    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd94e543-qh6tt"]
Nov 01 20:28:40.429469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:40.429434    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d14408-e902-4289-a2b7-875aa970c8e4" path="/var/lib/kubelet/pods/12d14408-e902-4289-a2b7-875aa970c8e4/volumes"
Nov 01 20:28:45.462364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.462329    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"]
Nov 01 20:28:45.463001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.462986    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ffb933f6-4d42-4351-89e1-f7ba574c74f8" containerName="revoke-keys"
Nov 01 20:28:45.463062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.463002    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb933f6-4d42-4351-89e1-f7ba574c74f8" containerName="revoke-keys"
Nov 01 20:28:45.463062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.463015    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bc4be621-8e5f-4a64-ba35-c73af038444c" containerName="revoke-keys"
Nov 01 20:28:45.463062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.463021    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4be621-8e5f-4a64-ba35-c73af038444c" containerName="revoke-keys"
Nov 01 20:28:45.463165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.463117    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4be621-8e5f-4a64-ba35-c73af038444c" containerName="revoke-keys"
Nov 01 20:28:45.463165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.463135    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb933f6-4d42-4351-89e1-f7ba574c74f8" containerName="revoke-keys"
Nov 01 20:28:45.467947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.467922    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.477548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.477517    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"]
Nov 01 20:28:45.478057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.478024    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:28:45.478203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.478061    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:28:45.478203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.478077    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:28:45.478203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.478100    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:28:45.618717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.618681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtgb\" (UniqueName: \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.619121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.618728    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.619121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.619033    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.720909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.720820    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.721083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.720944    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtgb\" (UniqueName: \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.721083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.720990    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.721541 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.721518    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.723599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.723573    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.728290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.728263    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtgb\" (UniqueName: \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\") pod \"maas-api-revoke-sub-58b28677-fk654\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:45.793622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:45.793583    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:46.081667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:46.081637    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"]
Nov 01 20:28:46.082035 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:28:46.082010    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd79b4e_95ab_411f_a458_ccf14672d544.slice/crio-e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60 WatchSource:0}: Error finding container e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60: Status 404 returned error can't find the container with id e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60
Nov 01 20:28:47.041625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:47.041591    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654" event={"ID":"2cd79b4e-95ab-411f-a458-ccf14672d544","Type":"ContainerStarted","Data":"423cf129323072d4c8b55eb1f5f5872fe727e399c70ae50a3e7058e9315a7bdf"}
Nov 01 20:28:47.042149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:47.041643    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654" event={"ID":"2cd79b4e-95ab-411f-a458-ccf14672d544","Type":"ContainerStarted","Data":"e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60"}
Nov 01 20:28:47.098051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:47.097983    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654" podStartSLOduration=2.097961454 podStartE2EDuration="2.097961454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:47.097950232 +0000 UTC m=+1814.819028911" watchObservedRunningTime="2026-11-01 20:28:47.097961454 +0000 UTC m=+1814.819040157"
Nov 01 20:28:48.050441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:48.050406    5041 generic.go:356] "Generic (PLEG): container finished" podID="2cd79b4e-95ab-411f-a458-ccf14672d544" containerID="423cf129323072d4c8b55eb1f5f5872fe727e399c70ae50a3e7058e9315a7bdf" exitCode=0
Nov 01 20:28:48.050895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:48.050492    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654" event={"ID":"2cd79b4e-95ab-411f-a458-ccf14672d544","Type":"ContainerDied","Data":"423cf129323072d4c8b55eb1f5f5872fe727e399c70ae50a3e7058e9315a7bdf"}
Nov 01 20:28:49.499476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.497386    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:49.616104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.616001    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\") pod \"2cd79b4e-95ab-411f-a458-ccf14672d544\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") "
Nov 01 20:28:49.616282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.616187    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\" (UniqueName: \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\") pod \"2cd79b4e-95ab-411f-a458-ccf14672d544\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") "
Nov 01 20:28:49.616282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.616223    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\") pod \"2cd79b4e-95ab-411f-a458-ccf14672d544\" (UID: \"2cd79b4e-95ab-411f-a458-ccf14672d544\") "
Nov 01 20:28:49.616819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.616771    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca" pod "2cd79b4e-95ab-411f-a458-ccf14672d544" (UID: "2cd79b4e-95ab-411f-a458-ccf14672d544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:49.619051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.619023    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca" pod "2cd79b4e-95ab-411f-a458-ccf14672d544" (UID: "2cd79b4e-95ab-411f-a458-ccf14672d544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:49.619183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.619092    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb" pod "2cd79b4e-95ab-411f-a458-ccf14672d544" (UID: "2cd79b4e-95ab-411f-a458-ccf14672d544"). InnerVolumeSpecName "kube-api-access-2vtgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:49.722831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.720313    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:49.722831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.720356    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd79b4e-95ab-411f-a458-ccf14672d544-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:49.722831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:49.720448    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtgb\" (UniqueName: \"kubernetes.io/projected/2cd79b4e-95ab-411f-a458-ccf14672d544-kube-api-access-2vtgb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:50.024116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.024029    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"]
Nov 01 20:28:50.027871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.027844    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b31bb19b-tvfgw"]
Nov 01 20:28:50.066964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.066935    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"
Nov 01 20:28:50.067163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.066962    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654" event={"ID":"2cd79b4e-95ab-411f-a458-ccf14672d544","Type":"ContainerDied","Data":"e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60"}
Nov 01 20:28:50.067163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.067010    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e926faef1fcb9c527e3b710dc1d426f2a8521a044e6a70deba3253053f571b60"
Nov 01 20:28:50.136596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.136554    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"]
Nov 01 20:28:50.137292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.137271    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2cd79b4e-95ab-411f-a458-ccf14672d544" containerName="revoke-keys"
Nov 01 20:28:50.137292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.137291    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd79b4e-95ab-411f-a458-ccf14672d544" containerName="revoke-keys"
Nov 01 20:28:50.137509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.137494    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd79b4e-95ab-411f-a458-ccf14672d544" containerName="revoke-keys"
Nov 01 20:28:50.141165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.141143    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.144583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.144559    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:28:50.150189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.150164    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"]
Nov 01 20:28:50.150480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.150459    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:28:50.150630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.150512    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:28:50.171443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.171403    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:28:50.226729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.226681    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.226982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.226956    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjx9\" (UniqueName: \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.227077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.227049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.328129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.328084    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.328350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.328231    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjx9\" (UniqueName: \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.328350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.328279    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.329296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.329240    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.330830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.330763    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.336095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.336076    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjx9\" (UniqueName: \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\") pod \"maas-api-revoke-sub-f43b6628-89d7s\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.431293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.431257    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ea781a-13ed-4ee9-99bd-d810a4f6efde" path="/var/lib/kubelet/pods/76ea781a-13ed-4ee9-99bd-d810a4f6efde/volumes"
Nov 01 20:28:50.457874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.457839    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:50.769947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:50.769916    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"]
Nov 01 20:28:50.771006 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:28:50.770979    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225629ac_15c9_4580_9c90_355fba578276.slice/crio-a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4 WatchSource:0}: Error finding container a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4: Status 404 returned error can't find the container with id a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4
Nov 01 20:28:51.075850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:51.075809    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s" event={"ID":"225629ac-15c9-4580-9c90-355fba578276","Type":"ContainerStarted","Data":"a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4"}
Nov 01 20:28:52.091959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:52.091916    5041 generic.go:356] "Generic (PLEG): container finished" podID="225629ac-15c9-4580-9c90-355fba578276" containerID="fa31e6c2833d03c49e8969f5cf5028467b03d7f1e04691ec554f44e1ca836c0e" exitCode=0
Nov 01 20:28:52.092366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:52.091964    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s" event={"ID":"225629ac-15c9-4580-9c90-355fba578276","Type":"ContainerDied","Data":"fa31e6c2833d03c49e8969f5cf5028467b03d7f1e04691ec554f44e1ca836c0e"}
Nov 01 20:28:53.028459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.028417    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" containerID="cri-o://fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7" gracePeriod=45
Nov 01 20:28:53.226955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.226908    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main" probeResult="failure" output="Get \"https://10.132.0.165:8000/ready\": dial tcp 10.132.0.165:8000: connect: connection refused"
Nov 01 20:28:53.298470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.298443    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:53.370376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.370341    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:28:53.460124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460091    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460237    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\") pod \"225629ac-15c9-4580-9c90-355fba578276\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") "
Nov 01 20:28:53.460321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\" (UniqueName: \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460316    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\") pod \"225629ac-15c9-4580-9c90-355fba578276\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") "
Nov 01 20:28:53.460512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460338    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460394    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460433    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\" (UniqueName: \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460512 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460411    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.460744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460540    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\" (UniqueName: \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\") pod \"225629ac-15c9-4580-9c90-355fba578276\" (UID: \"225629ac-15c9-4580-9c90-355fba578276\") "
Nov 01 20:28:53.460744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460605    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca" pod "225629ac-15c9-4580-9c90-355fba578276" (UID: "225629ac-15c9-4580-9c90-355fba578276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:28:53.460744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460612    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\") pod \"bcded9ee-c2f8-4780-9234-705481fb1333\" (UID: \"bcded9ee-c2f8-4780-9234-705481fb1333\") "
Nov 01 20:28:53.460744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460647    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.461000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.460843    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.461415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.461391    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225629ac-15c9-4580-9c90-355fba578276-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.461471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.461423    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.461471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.461438    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.461471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.461454    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.462744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.462722    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:28:53.463007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.462976    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca" pod "225629ac-15c9-4580-9c90-355fba578276" (UID: "225629ac-15c9-4580-9c90-355fba578276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:53.463112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.463092    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "kube-api-access-c8w8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:53.463161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.463112    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9" pod "225629ac-15c9-4580-9c90-355fba578276" (UID: "225629ac-15c9-4580-9c90-355fba578276"). InnerVolumeSpecName "kube-api-access-8rjx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:28:53.463161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.463113    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs" pod "bcded9ee-c2f8-4780-9234-705481fb1333" (UID: "bcded9ee-c2f8-4780-9234-705481fb1333"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:28:53.563135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.563051    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcded9ee-c2f8-4780-9234-705481fb1333-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.563135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.563082    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225629ac-15c9-4580-9c90-355fba578276-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.563135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.563092    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcded9ee-c2f8-4780-9234-705481fb1333-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.563135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.563101    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8w8z\" (UniqueName: \"kubernetes.io/projected/bcded9ee-c2f8-4780-9234-705481fb1333-kube-api-access-c8w8z\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:53.563135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:53.563113    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjx9\" (UniqueName: \"kubernetes.io/projected/225629ac-15c9-4580-9c90-355fba578276-kube-api-access-8rjx9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:28:54.109137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.109099    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"
Nov 01 20:28:54.109336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.109133    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s" event={"ID":"225629ac-15c9-4580-9c90-355fba578276","Type":"ContainerDied","Data":"a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4"}
Nov 01 20:28:54.109336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.109196    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2bae56d076141b11df30a890b83e2c3b3b4c5698c2a7560b65cc6541900a8d4"
Nov 01 20:28:54.110949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.110921    5041 generic.go:356] "Generic (PLEG): container finished" podID="bcded9ee-c2f8-4780-9234-705481fb1333" containerID="fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7" exitCode=0
Nov 01 20:28:54.111068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.110971    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"
Nov 01 20:28:54.111068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.110970    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" event={"ID":"bcded9ee-c2f8-4780-9234-705481fb1333","Type":"ContainerDied","Data":"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"}
Nov 01 20:28:54.111142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.111085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb" event={"ID":"bcded9ee-c2f8-4780-9234-705481fb1333","Type":"ContainerDied","Data":"c4e3a58e6f3f5320fe2c9d40b7e763b35abc24be6c6471cbeb5ac68b299fbb06"}
Nov 01 20:28:54.111142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.111105    5041 scope.go:122] "RemoveContainer" containerID="fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"
Nov 01 20:28:54.123217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.123191    5041 scope.go:122] "RemoveContainer" containerID="fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"
Nov 01 20:28:54.123602 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:28:54.123580    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7\": container with ID starting with fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7 not found: ID does not exist" containerID="fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"
Nov 01 20:28:54.123692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.123614    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7"} err="failed to get container status \"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7\": rpc error: code = NotFound desc = could not find container \"fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7\": container with ID starting with fff788023abfa2a2aa8550c26f3762d9e3e66a62a87b7636c283043e6aeb73d7 not found: ID does not exist"
Nov 01 20:28:54.140882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.140774    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:28:54.144879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.144854    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9de2ac/simulator-w3-9de2ac-kserve-744fbd68b8-mdhcb"]
Nov 01 20:28:54.429814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:28:54.429708    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" path="/var/lib/kubelet/pods/bcded9ee-c2f8-4780-9234-705481fb1333/volumes"
Nov 01 20:28:58.542627 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:28:58.542582    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:29:00.382711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.382672    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"]
Nov 01 20:29:00.383520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383500    5041 cpu_manager.go:414] "containerMap: removing container" podUID="225629ac-15c9-4580-9c90-355fba578276" containerName="revoke-keys"
Nov 01 20:29:00.383573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383522    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225629ac-15c9-4580-9c90-355fba578276" containerName="revoke-keys"
Nov 01 20:29:00.383573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383542    5041 cpu_manager.go:414] "containerMap: removing container" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main"
Nov 01 20:29:00.383573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383547    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main"
Nov 01 20:29:00.383674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383666    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="225629ac-15c9-4580-9c90-355fba578276" containerName="revoke-keys"
Nov 01 20:29:00.383709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.383678    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcded9ee-c2f8-4780-9234-705481fb1333" containerName="main"
Nov 01 20:29:00.388742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.388721    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.393103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.393067    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:29:00.393287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.393121    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:29:00.393761 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.393740    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:29:00.393928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.393816    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:29:00.395528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.395508    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"]
Nov 01 20:29:00.477981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.477946    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.478300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.478270    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdtv\" (UniqueName: \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.478346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.478318    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.580235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.580198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdtv\" (UniqueName: \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.580235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.580234    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.580460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.580284    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.581082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.581055    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.583076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.583050    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.587866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.587844    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdtv\" (UniqueName: \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\") pod \"maas-api-revoke-sub-08f2019b-7pcpp\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.708629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.708545    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:00.990527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:00.990499    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"]
Nov 01 20:29:00.992173 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:29:00.992139    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75187edf_276c_41ea_b61a_6609357ec1a0.slice/crio-7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69 WatchSource:0}: Error finding container 7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69: Status 404 returned error can't find the container with id 7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69
Nov 01 20:29:01.243617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:01.243541    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp" event={"ID":"75187edf-276c-41ea-b61a-6609357ec1a0","Type":"ContainerStarted","Data":"7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69"}
Nov 01 20:29:02.254611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:02.254569    5041 generic.go:356] "Generic (PLEG): container finished" podID="75187edf-276c-41ea-b61a-6609357ec1a0" containerID="9975eb95e5e38501b8242583c8ce4dc0cd00215a13e9b2fcdf93dfb57acba594" exitCode=0
Nov 01 20:29:02.255086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:02.254644    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp" event={"ID":"75187edf-276c-41ea-b61a-6609357ec1a0","Type":"ContainerDied","Data":"9975eb95e5e38501b8242583c8ce4dc0cd00215a13e9b2fcdf93dfb57acba594"}
Nov 01 20:29:04.283099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.281920    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:29:04.429055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.429019    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:04.603842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.603802    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\" (UniqueName: \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\") pod \"75187edf-276c-41ea-b61a-6609357ec1a0\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") "
Nov 01 20:29:04.604331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.604306    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\") pod \"75187edf-276c-41ea-b61a-6609357ec1a0\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") "
Nov 01 20:29:04.604903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.604881    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\") pod \"75187edf-276c-41ea-b61a-6609357ec1a0\" (UID: \"75187edf-276c-41ea-b61a-6609357ec1a0\") "
Nov 01 20:29:04.644528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.640315    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca" pod "75187edf-276c-41ea-b61a-6609357ec1a0" (UID: "75187edf-276c-41ea-b61a-6609357ec1a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:04.671414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.671377    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv" pod "75187edf-276c-41ea-b61a-6609357ec1a0" (UID: "75187edf-276c-41ea-b61a-6609357ec1a0"). InnerVolumeSpecName "kube-api-access-nmdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:04.680075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.680044    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:04.680304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.680285    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdtv\" (UniqueName: \"kubernetes.io/projected/75187edf-276c-41ea-b61a-6609357ec1a0-kube-api-access-nmdtv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:04.778802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.768926    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca" pod "75187edf-276c-41ea-b61a-6609357ec1a0" (UID: "75187edf-276c-41ea-b61a-6609357ec1a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:29:04.784801 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.782182    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75187edf-276c-41ea-b61a-6609357ec1a0-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:04.797939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.797904    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"
Nov 01 20:29:04.798243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.798217    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp" event={"ID":"75187edf-276c-41ea-b61a-6609357ec1a0","Type":"ContainerDied","Data":"7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69"}
Nov 01 20:29:04.798431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:04.798413    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ffc713bbb55fd38804b49763834856f7c4dbac56f40af2612653896e3564c69"
Nov 01 20:29:05.126009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:05.125963    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:29:05.286601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:05.286558    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5" podUID="5a81bba4-497d-453e-844a-a9e5cbc1f94b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:29:05.287896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:05.287852    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:29:05.311576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:05.311537    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5"
Nov 01 20:29:11.042396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.042361    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"]
Nov 01 20:29:11.042998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.042982    5041 cpu_manager.go:414] "containerMap: removing container" podUID="75187edf-276c-41ea-b61a-6609357ec1a0" containerName="revoke-keys"
Nov 01 20:29:11.043066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.042999    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75187edf-276c-41ea-b61a-6609357ec1a0" containerName="revoke-keys"
Nov 01 20:29:11.043111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.043103    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="75187edf-276c-41ea-b61a-6609357ec1a0" containerName="revoke-keys"
Nov 01 20:29:11.047995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.047978    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.056890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.056871    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:29:11.057993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.057071    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:29:11.057993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.057149    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:29:11.057993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.057204    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:29:11.061827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.061801    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"]
Nov 01 20:29:11.141912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.141869    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4rp\" (UniqueName: \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.142091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.142045    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.142142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.142093    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.243431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.243387    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.243634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.243444    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.243971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.243640    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4rp\" (UniqueName: \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.244137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.244115    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.245851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.245828    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.251284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.251265    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4rp\" (UniqueName: \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\") pod \"maas-api-revoke-sub-f35b6611-vbwbm\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.363932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.363853    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:11.652619 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:29:11.652590    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1530d04e_5598_4f5b_9aa6_e9f2754322e5.slice/crio-273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da WatchSource:0}: Error finding container 273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da: Status 404 returned error can't find the container with id 273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da
Nov 01 20:29:11.652971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.652947    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"]
Nov 01 20:29:11.866746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:11.866711    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm" event={"ID":"1530d04e-5598-4f5b-9aa6-e9f2754322e5","Type":"ContainerStarted","Data":"273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da"}
Nov 01 20:29:12.878477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:12.878384    5041 generic.go:356] "Generic (PLEG): container finished" podID="1530d04e-5598-4f5b-9aa6-e9f2754322e5" containerID="b02de434d73b5d00505422f2165c7aca21e0255731531c5c75818daff7e56e69" exitCode=0
Nov 01 20:29:12.879023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:12.878508    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm" event={"ID":"1530d04e-5598-4f5b-9aa6-e9f2754322e5","Type":"ContainerDied","Data":"b02de434d73b5d00505422f2165c7aca21e0255731531c5c75818daff7e56e69"}
Nov 01 20:29:14.056303 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.056274    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:14.079433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.079402    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\" (UniqueName: \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\") pod \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") "
Nov 01 20:29:14.079624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.079532    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\") pod \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") "
Nov 01 20:29:14.079698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.079641    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\") pod \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\" (UID: \"1530d04e-5598-4f5b-9aa6-e9f2754322e5\") "
Nov 01 20:29:14.080385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.080360    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca" pod "1530d04e-5598-4f5b-9aa6-e9f2754322e5" (UID: "1530d04e-5598-4f5b-9aa6-e9f2754322e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:14.093967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.093933    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca" pod "1530d04e-5598-4f5b-9aa6-e9f2754322e5" (UID: "1530d04e-5598-4f5b-9aa6-e9f2754322e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:29:14.094095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.093984    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp" pod "1530d04e-5598-4f5b-9aa6-e9f2754322e5" (UID: "1530d04e-5598-4f5b-9aa6-e9f2754322e5"). InnerVolumeSpecName "kube-api-access-5b4rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:14.182182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.182001    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4rp\" (UniqueName: \"kubernetes.io/projected/1530d04e-5598-4f5b-9aa6-e9f2754322e5-kube-api-access-5b4rp\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:14.182432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.182257    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:14.182432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.182283    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1530d04e-5598-4f5b-9aa6-e9f2754322e5-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:14.492522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.492489    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:29:14.492762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.492723    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" containerID="cri-o://d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18" gracePeriod=30
Nov 01 20:29:14.895900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.895871    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"
Nov 01 20:29:14.896092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.895909    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm" event={"ID":"1530d04e-5598-4f5b-9aa6-e9f2754322e5","Type":"ContainerDied","Data":"273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da"}
Nov 01 20:29:14.896092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:14.895969    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="273ffb914ee52f8c66e9ea8a63f5c5f7c72b2a98a83c34a5dd312ec28dd9e7da"
Nov 01 20:29:19.466308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.466270    5041 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:29:19.466715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.466342    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:29:19.769021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.768989    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:29:19.867747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.867710    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.867928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.867759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.867928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.867829    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.867928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.867904    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.867937    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868037    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868166    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868188    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:29:19.868344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868234    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:29:19.868344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868266    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:29:19.868344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868315    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868377    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\") pod \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\" (UID: \"71cc6edf-12d2-41f7-805d-5fa131a17f3e\") "
Nov 01 20:29:19.868607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868577    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:29:19.868713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.868584    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:19.869083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.869060    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.869083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.869084    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.869347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.869099    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.869347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.869113    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.869347 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.869125    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.870406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.870384    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:29:19.870741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.870718    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:29:19.871005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.870985    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "kube-api-access-8kcxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:19.871005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.870998    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token" pod "71cc6edf-12d2-41f7-805d-5fa131a17f3e" (UID: "71cc6edf-12d2-41f7-805d-5fa131a17f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:19.943209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.943168    5041 generic.go:356] "Generic (PLEG): container finished" podID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerID="d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18" exitCode=0
Nov 01 20:29:19.943410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.943231    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"
Nov 01 20:29:19.943410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.943254    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerDied","Data":"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"}
Nov 01 20:29:19.943410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.943339    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql" event={"ID":"71cc6edf-12d2-41f7-805d-5fa131a17f3e","Type":"ContainerDied","Data":"b77783bbd8ba3d2fe6d00e83aabaa0cccde27558abb27db507be7ea08a6a5085"}
Nov 01 20:29:19.943410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.943380    5041 scope.go:122] "RemoveContainer" containerID="d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"
Nov 01 20:29:19.955584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.955562    5041 scope.go:122] "RemoveContainer" containerID="172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"
Nov 01 20:29:19.966292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.966269    5041 scope.go:122] "RemoveContainer" containerID="d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"
Nov 01 20:29:19.966601 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:29:19.966579    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18\": container with ID starting with d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18 not found: ID does not exist" containerID="d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"
Nov 01 20:29:19.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.966610    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18"} err="failed to get container status \"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18\": rpc error: code = NotFound desc = could not find container \"d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18\": container with ID starting with d1d4a66c2a21a490567a3d3151b9f7c22e1dbeca6feaa203e7740beb9583bd18 not found: ID does not exist"
Nov 01 20:29:19.966688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.966628    5041 scope.go:122] "RemoveContainer" containerID="172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"
Nov 01 20:29:19.966966 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:29:19.966941    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39\": container with ID starting with 172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39 not found: ID does not exist" containerID="172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"
Nov 01 20:29:19.967038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.966976    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39"} err="failed to get container status \"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39\": rpc error: code = NotFound desc = could not find container \"172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39\": container with ID starting with 172a4e52940d3f12ffefe1868b8a1e23ba5f0c72e7bb1f6a9ee7c3807f47dd39 not found: ID does not exist"
Nov 01 20:29:19.968814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.968790    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:29:19.970372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.970344    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-kube-api-access-8kcxk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.970372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.970373    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.970505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.970386    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.970505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.970399    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71cc6edf-12d2-41f7-805d-5fa131a17f3e-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:19.973287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:19.973267    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-lt9ql"]
Nov 01 20:29:20.429847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:20.429803    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" path="/var/lib/kubelet/pods/71cc6edf-12d2-41f7-805d-5fa131a17f3e/volumes"
Nov 01 20:29:21.372607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.372571    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"]
Nov 01 20:29:21.373414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373392    5041 cpu_manager.go:414] "containerMap: removing container" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.373521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373417    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.373521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373438    5041 cpu_manager.go:414] "containerMap: removing container" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.373521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373447    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.373521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373518    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1530d04e-5598-4f5b-9aa6-e9f2754322e5" containerName="revoke-keys"
Nov 01 20:29:21.373744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373527    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1530d04e-5598-4f5b-9aa6-e9f2754322e5" containerName="revoke-keys"
Nov 01 20:29:21.373744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373680    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1530d04e-5598-4f5b-9aa6-e9f2754322e5" containerName="revoke-keys"
Nov 01 20:29:21.373744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373694    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.373744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.373709    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cc6edf-12d2-41f7-805d-5fa131a17f3e" containerName="istio-proxy"
Nov 01 20:29:21.379030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.379007    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.382608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.382587    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:29:21.383517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.383401    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:29:21.383517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.383447    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:29:21.383517 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.383494    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:29:21.384010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.383992    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"]
Nov 01 20:29:21.489282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.489245    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw8f\" (UniqueName: \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.489473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.489449    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.489584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.489568    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.591021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.590981    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.591211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.591091    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.591211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.591179    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw8f\" (UniqueName: \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.591724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.591698    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.593447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.593425    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.599226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.599193    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw8f\" (UniqueName: \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\") pod \"maas-api-revoke-sub-38ed5f29-ltxwv\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.697135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.697061    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:21.982502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:21.982472    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"]
Nov 01 20:29:21.982646 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:29:21.982480    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13621076_3035_4dd2_950d_997e1e84340c.slice/crio-efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392 WatchSource:0}: Error finding container efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392: Status 404 returned error can't find the container with id efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392
Nov 01 20:29:22.969043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:22.969006    5041 generic.go:356] "Generic (PLEG): container finished" podID="13621076-3035-4dd2-950d-997e1e84340c" containerID="df6f0bb6d920f9c2edd4505fdcec052c5aca9ee29f30b476faccb9ce58ca6161" exitCode=0
Nov 01 20:29:22.969446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:22.969085    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv" event={"ID":"13621076-3035-4dd2-950d-997e1e84340c","Type":"ContainerDied","Data":"df6f0bb6d920f9c2edd4505fdcec052c5aca9ee29f30b476faccb9ce58ca6161"}
Nov 01 20:29:22.969446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:22.969130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv" event={"ID":"13621076-3035-4dd2-950d-997e1e84340c","Type":"ContainerStarted","Data":"efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392"}
Nov 01 20:29:24.156525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.156500    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:24.232142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.232043    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\" (UniqueName: \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\") pod \"13621076-3035-4dd2-950d-997e1e84340c\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") "
Nov 01 20:29:24.232312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.232174    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\") pod \"13621076-3035-4dd2-950d-997e1e84340c\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") "
Nov 01 20:29:24.232366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.232318    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\") pod \"13621076-3035-4dd2-950d-997e1e84340c\" (UID: \"13621076-3035-4dd2-950d-997e1e84340c\") "
Nov 01 20:29:24.232652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.232625    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca" pod "13621076-3035-4dd2-950d-997e1e84340c" (UID: "13621076-3035-4dd2-950d-997e1e84340c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:29:24.232981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.232960    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13621076-3035-4dd2-950d-997e1e84340c-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:24.234219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.234191    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca" pod "13621076-3035-4dd2-950d-997e1e84340c" (UID: "13621076-3035-4dd2-950d-997e1e84340c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:29:24.234339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.234316    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f" pod "13621076-3035-4dd2-950d-997e1e84340c" (UID: "13621076-3035-4dd2-950d-997e1e84340c"). InnerVolumeSpecName "kube-api-access-kgw8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:29:24.334198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.334166    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgw8f\" (UniqueName: \"kubernetes.io/projected/13621076-3035-4dd2-950d-997e1e84340c-kube-api-access-kgw8f\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:24.334198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.334197    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13621076-3035-4dd2-950d-997e1e84340c-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:29:24.986475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.986440    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"
Nov 01 20:29:24.986642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.986473    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv" event={"ID":"13621076-3035-4dd2-950d-997e1e84340c","Type":"ContainerDied","Data":"efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392"}
Nov 01 20:29:24.986642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:24.986536    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efebd86952307a79489a798a9af71deee6b212ae0f506e200fa35d55624a2392"
Nov 01 20:29:28.542424 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:29:28.542383    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:29:36.058211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:36.058179    5041 scope.go:122] "RemoveContainer" containerID="f5ef33196764fcfe3601f3690a6c89fd95106bed9b86a12349afdf98b35db892"
Nov 01 20:29:42.023288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:42.023255    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"]
Nov 01 20:29:42.027291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:42.027267    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f258d1ff-tdmxf"]
Nov 01 20:29:42.429242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:42.429210    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a945cc4-b103-4b78-9224-96cf9ae37294" path="/var/lib/kubelet/pods/4a945cc4-b103-4b78-9224-96cf9ae37294/volumes"
Nov 01 20:29:53.021332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:53.021300    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"]
Nov 01 20:29:53.048464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:53.048393    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21d59845-g8zlh"]
Nov 01 20:29:54.429114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:54.429080    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab6debb-63fa-43d6-a127-6b1d8e43c1f8" path="/var/lib/kubelet/pods/cab6debb-63fa-43d6-a127-6b1d8e43c1f8/volumes"
Nov 01 20:29:58.020446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:58.020403    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"]
Nov 01 20:29:58.024120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:58.024098    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dc5816-mb6kd"]
Nov 01 20:29:58.429479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:29:58.429441    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6432505-25f6-4a47-b482-4b7a2d64d582" path="/var/lib/kubelet/pods/e6432505-25f6-4a47-b482-4b7a2d64d582/volumes"
Nov 01 20:29:58.543003 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:29:58.542970    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:30:00.142247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.142214    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:00.142909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.142891    5041 cpu_manager.go:414] "containerMap: removing container" podUID="13621076-3035-4dd2-950d-997e1e84340c" containerName="revoke-keys"
Nov 01 20:30:00.142969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.142910    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13621076-3035-4dd2-950d-997e1e84340c" containerName="revoke-keys"
Nov 01 20:30:00.143052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.143042    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="13621076-3035-4dd2-950d-997e1e84340c" containerName="revoke-keys"
Nov 01 20:30:00.146703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.146682    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:00.146854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.146831    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.150677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.150651    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.151023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.151003    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:00.156101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.155945    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.159749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.159423    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:30:00.160354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.160331    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:30:00.160463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.160428    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:30:00.160463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.160458    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:00.165649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.165624    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.165649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.165636    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:00.172548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.171319    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.172548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.171514    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:00.177899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.177872    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:00.187485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.185906    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:00.193944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.193908    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:00.212196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.211941    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:00.257567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.257524    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:00.265906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.265877    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:00.266165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.265993    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.274887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.274861    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:00.276256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.276225    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.280802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.280764    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.287758    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.287826    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtnv\" (UniqueName: \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.287898    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwprg\" (UniqueName: \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.287925    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288031    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsd7t\" (UniqueName: \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288080    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b2l\" (UniqueName: \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288136    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288191    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288255    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv88\" (UniqueName: \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.288828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.288328    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.293687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.293665    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:00.301002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.300976    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:00.306746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.306720    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:00.310933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.310907    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:00.316188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.316146    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.319457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.319436    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:00.390591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390551    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsd7t\" (UniqueName: \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.390814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390605    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b2l\" (UniqueName: \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.390814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390627    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.390931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390813    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.390931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390854    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpt2\" (UniqueName: \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.390931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390918    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.391070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.390982    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.391070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391057    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.391148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391104    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv88\" (UniqueName: \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.391217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391201    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.391290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391272    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgks5\" (UniqueName: \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.391348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391330    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.391408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391357    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtnv\" (UniqueName: \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.391468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391412    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.391468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391442    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwprg\" (UniqueName: \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.391583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391469    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.391647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391598    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.391647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391603    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdcl\" (UniqueName: \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.391766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391668    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6wwv\" (UniqueName: \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.391766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.391672    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.392065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.392045    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.392141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.392057    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.392141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.392125    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.399680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.399624    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv88\" (UniqueName: \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\") pod \"maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.399680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.399637    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsd7t\" (UniqueName: \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.399852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.399706    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b2l\" (UniqueName: \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\") pod \"maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.400119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.400097    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtnv\" (UniqueName: \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.400668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.400652    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwprg\" (UniqueName: \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.472824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.472789    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:00.488058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.488035    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:00.492747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.492718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgks5\" (UniqueName: \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.492895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.492870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.492988 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.492966    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdcl\" (UniqueName: \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.493061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.493013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6wwv\" (UniqueName: \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.493202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.493178    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.493344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.493277    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpt2\" (UniqueName: \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.493922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.493898    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.494114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.494092    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.494626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.494602    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.494705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.494663    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.495315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.495273    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.495421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.495394    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.503201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.503173    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6wwv\" (UniqueName: \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\") pod \"maas-api-key-cleanup-29892750-fsrpf\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.503461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.503440    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpt2\" (UniqueName: \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.504190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.504112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdcl\" (UniqueName: \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\") pod \"maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.504275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.504253    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgks5\" (UniqueName: \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\") pod \"maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.519094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.519059    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:00.555317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.555080    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:00.582050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.581827    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:00.602217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.601980    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:00.611864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.611587    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:00.644051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.643821    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:00.645272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.645238    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:00.777439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.776583    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:00.994075 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:00.994046    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa101d42_7dc9_4f51_9997_77166e0bcc98.slice/crio-5c667cbcc27406403aaa8cd1b516e5102289a469fcf9345d729a83f8a4cebd83 WatchSource:0}: Error finding container 5c667cbcc27406403aaa8cd1b516e5102289a469fcf9345d729a83f8a4cebd83: Status 404 returned error can't find the container with id 5c667cbcc27406403aaa8cd1b516e5102289a469fcf9345d729a83f8a4cebd83
Nov 01 20:30:00.997841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:00.997122    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:01.049428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.049348    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:01.051910 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.051873    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f7c3ee_5e67_419d_ba33_0d7221a2d1eb.slice/crio-785551fbf9464f6215fc72d5602d1b9a83a7b4ae253fdc8045b11c119201903b WatchSource:0}: Error finding container 785551fbf9464f6215fc72d5602d1b9a83a7b4ae253fdc8045b11c119201903b: Status 404 returned error can't find the container with id 785551fbf9464f6215fc72d5602d1b9a83a7b4ae253fdc8045b11c119201903b
Nov 01 20:30:01.055476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.055401    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:01.066883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.065733    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:01.080030 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.079990    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279d7000_3813_4aff_bff0_f2d0753a00ee.slice/crio-96bb6266b900cf344fa45e2d918358e6004cac4bcd70b5b34c9c60d05c2ee184 WatchSource:0}: Error finding container 96bb6266b900cf344fa45e2d918358e6004cac4bcd70b5b34c9c60d05c2ee184: Status 404 returned error can't find the container with id 96bb6266b900cf344fa45e2d918358e6004cac4bcd70b5b34c9c60d05c2ee184
Nov 01 20:30:01.094814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.094751    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:01.108571 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.108524    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a780fc_d603_45e1_a958_e9d6b6ee4715.slice/crio-f6839bf4c18d20d37f5e62544c98b663a49e7cddd672844612dfa0bea8f74bbd WatchSource:0}: Error finding container f6839bf4c18d20d37f5e62544c98b663a49e7cddd672844612dfa0bea8f74bbd: Status 404 returned error can't find the container with id f6839bf4c18d20d37f5e62544c98b663a49e7cddd672844612dfa0bea8f74bbd
Nov 01 20:30:01.153533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.153503    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:01.181082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.181051    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:01.196288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.196255    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:01.257598 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.257562    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a0cea_2aee_436d_993e_1e7d302da192.slice/crio-0fd89898dfe6638243ff42f64cb9a4803160e013d1f24255b5fc642cc236b346 WatchSource:0}: Error finding container 0fd89898dfe6638243ff42f64cb9a4803160e013d1f24255b5fc642cc236b346: Status 404 returned error can't find the container with id 0fd89898dfe6638243ff42f64cb9a4803160e013d1f24255b5fc642cc236b346
Nov 01 20:30:01.258104 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.258081    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaef7d46_7470_41bf_897c_b181d3f94ff0.slice/crio-86bfd58ae102623e32f3e668fb7d6fc172c409eb927782c5512f5f859a78f898 WatchSource:0}: Error finding container 86bfd58ae102623e32f3e668fb7d6fc172c409eb927782c5512f5f859a78f898: Status 404 returned error can't find the container with id 86bfd58ae102623e32f3e668fb7d6fc172c409eb927782c5512f5f859a78f898
Nov 01 20:30:01.258646 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:01.258613    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318364b7_2ac9_4888_a2ac_599131e38f74.slice/crio-16224d7f2ed1fd1e93988b87feb75e2c55245d9174473864440f93d3bcf742c0 WatchSource:0}: Error finding container 16224d7f2ed1fd1e93988b87feb75e2c55245d9174473864440f93d3bcf742c0: Status 404 returned error can't find the container with id 16224d7f2ed1fd1e93988b87feb75e2c55245d9174473864440f93d3bcf742c0
Nov 01 20:30:01.311079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.311043    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerStarted","Data":"c00155003d28a8c76aa420b3642d4d2c900f4bdb65d5a2ab9c190b5e8cc13716"}
Nov 01 20:30:01.313051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.313021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerStarted","Data":"86bfd58ae102623e32f3e668fb7d6fc172c409eb927782c5512f5f859a78f898"}
Nov 01 20:30:01.314691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.314656    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" event={"ID":"fa101d42-7dc9-4f51-9997-77166e0bcc98","Type":"ContainerStarted","Data":"5c667cbcc27406403aaa8cd1b516e5102289a469fcf9345d729a83f8a4cebd83"}
Nov 01 20:30:01.316211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.316184    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerStarted","Data":"0fd89898dfe6638243ff42f64cb9a4803160e013d1f24255b5fc642cc236b346"}
Nov 01 20:30:01.317928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.317901    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerStarted","Data":"c9a06d8ee3d6a01b993c1062afb4fd5cc23a05836c720a8457661b0ca14a6487"}
Nov 01 20:30:01.319260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.319233    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" event={"ID":"26a780fc-d603-45e1-a958-e9d6b6ee4715","Type":"ContainerStarted","Data":"f6839bf4c18d20d37f5e62544c98b663a49e7cddd672844612dfa0bea8f74bbd"}
Nov 01 20:30:01.320578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.320548    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerStarted","Data":"16224d7f2ed1fd1e93988b87feb75e2c55245d9174473864440f93d3bcf742c0"}
Nov 01 20:30:01.322035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.322014    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerStarted","Data":"96bb6266b900cf344fa45e2d918358e6004cac4bcd70b5b34c9c60d05c2ee184"}
Nov 01 20:30:01.323416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:01.323392    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerStarted","Data":"785551fbf9464f6215fc72d5602d1b9a83a7b4ae253fdc8045b11c119201903b"}
Nov 01 20:30:02.334543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.334510    5041 generic.go:356] "Generic (PLEG): container finished" podID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerID="23af230fb3fa3684c8f8ad674e78de8a4251b2ddbe53c46317c6624e229fddaf" exitCode=60
Nov 01 20:30:02.335026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.334608    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" event={"ID":"fa101d42-7dc9-4f51-9997-77166e0bcc98","Type":"ContainerDied","Data":"23af230fb3fa3684c8f8ad674e78de8a4251b2ddbe53c46317c6624e229fddaf"}
Nov 01 20:30:02.335026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.334927    5041 scope.go:122] "RemoveContainer" containerID="23af230fb3fa3684c8f8ad674e78de8a4251b2ddbe53c46317c6624e229fddaf"
Nov 01 20:30:02.336495 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.336467    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerStarted","Data":"9eac58665373858a6e8ed303314420ca41daceda3bb066c4ee610d5d8730e3d6"}
Nov 01 20:30:02.338376 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.338351    5041 generic.go:356] "Generic (PLEG): container finished" podID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerID="53be26e925414ab2dd5ed42968d208507174c780d76c0b89095954cc1562647b" exitCode=60
Nov 01 20:30:02.338476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.338434    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerDied","Data":"53be26e925414ab2dd5ed42968d208507174c780d76c0b89095954cc1562647b"}
Nov 01 20:30:02.338631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.338616    5041 scope.go:122] "RemoveContainer" containerID="53be26e925414ab2dd5ed42968d208507174c780d76c0b89095954cc1562647b"
Nov 01 20:30:02.340199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.340178    5041 generic.go:356] "Generic (PLEG): container finished" podID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerID="92666a7f2501837cd84b3dc0ecab8b870e6bfebac601d60a349bb00c4f05c6bf" exitCode=60
Nov 01 20:30:02.340298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.340206    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" event={"ID":"26a780fc-d603-45e1-a958-e9d6b6ee4715","Type":"ContainerDied","Data":"92666a7f2501837cd84b3dc0ecab8b870e6bfebac601d60a349bb00c4f05c6bf"}
Nov 01 20:30:02.340472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.340446    5041 scope.go:122] "RemoveContainer" containerID="92666a7f2501837cd84b3dc0ecab8b870e6bfebac601d60a349bb00c4f05c6bf"
Nov 01 20:30:02.342107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.342080    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerStarted","Data":"53214c627825eb88da8ca9d525029cec4f2d4a08eca55542562bdfe52e8d0a26"}
Nov 01 20:30:02.344403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.344376    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerStarted","Data":"1e89759fab8263942feb447e41a456196b510f5afdecea2106b11917522a51ab"}
Nov 01 20:30:02.346531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.346507    5041 generic.go:356] "Generic (PLEG): container finished" podID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerID="edd5ce69cb4a294b6facfcfa6521d8b07be83610910f3c30988df70fce446005" exitCode=60
Nov 01 20:30:02.346661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.346583    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerDied","Data":"edd5ce69cb4a294b6facfcfa6521d8b07be83610910f3c30988df70fce446005"}
Nov 01 20:30:02.346869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.346847    5041 scope.go:122] "RemoveContainer" containerID="edd5ce69cb4a294b6facfcfa6521d8b07be83610910f3c30988df70fce446005"
Nov 01 20:30:02.351893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.350279    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerID="a4528085a8c07e561073597842415188d052f4a4271478fc1c5c379ae49cb280" exitCode=60
Nov 01 20:30:02.351893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.350414    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerDied","Data":"a4528085a8c07e561073597842415188d052f4a4271478fc1c5c379ae49cb280"}
Nov 01 20:30:02.351893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.350720    5041 scope.go:122] "RemoveContainer" containerID="a4528085a8c07e561073597842415188d052f4a4271478fc1c5c379ae49cb280"
Nov 01 20:30:02.353987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.353963    5041 generic.go:356] "Generic (PLEG): container finished" podID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerID="f6ccffdaba922e7fa641a6cd8df68a40722aad49d6f992b76db09607a7a62f44" exitCode=60
Nov 01 20:30:02.354112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.354015    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerDied","Data":"f6ccffdaba922e7fa641a6cd8df68a40722aad49d6f992b76db09607a7a62f44"}
Nov 01 20:30:02.354309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.354293    5041 scope.go:122] "RemoveContainer" containerID="f6ccffdaba922e7fa641a6cd8df68a40722aad49d6f992b76db09607a7a62f44"
Nov 01 20:30:02.374302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.374253    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" podStartSLOduration=2.374238588 podStartE2EDuration="2.374238588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.370622565 +0000 UTC m=+1890.091701242" watchObservedRunningTime="2026-11-01 20:30:02.374238588 +0000 UTC m=+1890.095317329"
Nov 01 20:30:02.390021 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.389962    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" podStartSLOduration=2.389944914 podStartE2EDuration="2.389944914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.38548175 +0000 UTC m=+1890.106560427" watchObservedRunningTime="2026-11-01 20:30:02.389944914 +0000 UTC m=+1890.111023591"
Nov 01 20:30:02.401591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:02.401497    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" podStartSLOduration=2.401481841 podStartE2EDuration="2.401481841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:02.399315679 +0000 UTC m=+1890.120394367" watchObservedRunningTime="2026-11-01 20:30:02.401481841 +0000 UTC m=+1890.122560545"
Nov 01 20:30:03.372497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.372459    5041 generic.go:356] "Generic (PLEG): container finished" podID="318364b7-2ac9-4888-a2ac-599131e38f74" containerID="53214c627825eb88da8ca9d525029cec4f2d4a08eca55542562bdfe52e8d0a26" exitCode=60
Nov 01 20:30:03.373007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.372549    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerDied","Data":"53214c627825eb88da8ca9d525029cec4f2d4a08eca55542562bdfe52e8d0a26"}
Nov 01 20:30:03.373252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.373228    5041 scope.go:122] "RemoveContainer" containerID="53214c627825eb88da8ca9d525029cec4f2d4a08eca55542562bdfe52e8d0a26"
Nov 01 20:30:03.375086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.375057    5041 generic.go:356] "Generic (PLEG): container finished" podID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerID="1e89759fab8263942feb447e41a456196b510f5afdecea2106b11917522a51ab" exitCode=60
Nov 01 20:30:03.375207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.375156    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerDied","Data":"1e89759fab8263942feb447e41a456196b510f5afdecea2106b11917522a51ab"}
Nov 01 20:30:03.375424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.375406    5041 scope.go:122] "RemoveContainer" containerID="1e89759fab8263942feb447e41a456196b510f5afdecea2106b11917522a51ab"
Nov 01 20:30:03.378344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.378318    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerStarted","Data":"db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"}
Nov 01 20:30:03.380615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.380590    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerStarted","Data":"45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"}
Nov 01 20:30:03.382632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.382608    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerStarted","Data":"a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"}
Nov 01 20:30:03.386620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.386593    5041 generic.go:356] "Generic (PLEG): container finished" podID="280a0cea-2aee-436d-993e-1e7d302da192" containerID="9eac58665373858a6e8ed303314420ca41daceda3bb066c4ee610d5d8730e3d6" exitCode=60
Nov 01 20:30:03.386708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.386665    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerDied","Data":"9eac58665373858a6e8ed303314420ca41daceda3bb066c4ee610d5d8730e3d6"}
Nov 01 20:30:03.386934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.386916    5041 scope.go:122] "RemoveContainer" containerID="9eac58665373858a6e8ed303314420ca41daceda3bb066c4ee610d5d8730e3d6"
Nov 01 20:30:03.388709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.388690    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerStarted","Data":"c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"}
Nov 01 20:30:03.410595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.409909    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" podStartSLOduration=3.409892194 podStartE2EDuration="3.409892194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.405539077 +0000 UTC m=+1891.126617754" watchObservedRunningTime="2026-11-01 20:30:03.409892194 +0000 UTC m=+1891.130970871"
Nov 01 20:30:03.429526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.429471    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" podStartSLOduration=3.429452866 podStartE2EDuration="3.429452866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.424253619 +0000 UTC m=+1891.145332301" watchObservedRunningTime="2026-11-01 20:30:03.429452866 +0000 UTC m=+1891.150531543"
Nov 01 20:30:03.440744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.440694    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" podStartSLOduration=3.440680157 podStartE2EDuration="3.440680157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.438033282 +0000 UTC m=+1891.159111958" watchObservedRunningTime="2026-11-01 20:30:03.440680157 +0000 UTC m=+1891.161758832"
Nov 01 20:30:03.456586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.456541    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" podStartSLOduration=3.456525575 podStartE2EDuration="3.456525575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.451709883 +0000 UTC m=+1891.172788560" watchObservedRunningTime="2026-11-01 20:30:03.456525575 +0000 UTC m=+1891.177604308"
Nov 01 20:30:03.467891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:03.467838    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" podStartSLOduration=3.467747684 podStartE2EDuration="3.467747684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:03.465721493 +0000 UTC m=+1891.186800169" watchObservedRunningTime="2026-11-01 20:30:03.467747684 +0000 UTC m=+1891.188826428"
Nov 01 20:30:04.399377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.399340    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerID="45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5" exitCode=60
Nov 01 20:30:04.399956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.399476    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerDied","Data":"45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"}
Nov 01 20:30:04.399956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.399533    5041 scope.go:122] "RemoveContainer" containerID="a4528085a8c07e561073597842415188d052f4a4271478fc1c5c379ae49cb280"
Nov 01 20:30:04.399956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.399873    5041 scope.go:122] "RemoveContainer" containerID="45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"
Nov 01 20:30:04.400250 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.400228    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf_odh-ai-gateway-infra(1fc3d458-db41-4e30-8cd3-c8280fb488f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0"
Nov 01 20:30:04.402975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.402946    5041 generic.go:356] "Generic (PLEG): container finished" podID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerID="a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009" exitCode=60
Nov 01 20:30:04.403115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.402982    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerDied","Data":"a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"}
Nov 01 20:30:04.403344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.403321    5041 scope.go:122] "RemoveContainer" containerID="a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"
Nov 01 20:30:04.403767 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.403690    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm_odh-ai-gateway-infra(caef7d46-7470-41bf-897c-b181d3f94ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0"
Nov 01 20:30:04.405288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.405266    5041 generic.go:356] "Generic (PLEG): container finished" podID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerID="e4eee5b1841128b33b21324e6283274bbabd5f227b969d5ad66973583048cdcb" exitCode=60
Nov 01 20:30:04.405386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.405289    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" event={"ID":"fa101d42-7dc9-4f51-9997-77166e0bcc98","Type":"ContainerDied","Data":"e4eee5b1841128b33b21324e6283274bbabd5f227b969d5ad66973583048cdcb"}
Nov 01 20:30:04.406070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.405632    5041 scope.go:122] "RemoveContainer" containerID="e4eee5b1841128b33b21324e6283274bbabd5f227b969d5ad66973583048cdcb"
Nov 01 20:30:04.406070 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.406028    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx_odh-ai-gateway-infra(fa101d42-7dc9-4f51-9997-77166e0bcc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98"
Nov 01 20:30:04.407454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.407429    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerStarted","Data":"1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"}
Nov 01 20:30:04.409662 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.409638    5041 generic.go:356] "Generic (PLEG): container finished" podID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerID="c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96" exitCode=60
Nov 01 20:30:04.409751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.409738    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerDied","Data":"c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"}
Nov 01 20:30:04.410035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.410013    5041 scope.go:122] "RemoveContainer" containerID="c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"
Nov 01 20:30:04.410272 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.410254    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl_odh-ai-gateway-infra(8aaf972e-adcb-492c-b944-0a17205f31e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2"
Nov 01 20:30:04.411670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.411649    5041 generic.go:356] "Generic (PLEG): container finished" podID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerID="7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e" exitCode=60
Nov 01 20:30:04.411824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.411770    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" event={"ID":"26a780fc-d603-45e1-a958-e9d6b6ee4715","Type":"ContainerDied","Data":"7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e"}
Nov 01 20:30:04.412036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.412016    5041 scope.go:122] "RemoveContainer" containerID="7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e"
Nov 01 20:30:04.412277 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.412254    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d_odh-ai-gateway-infra(26a780fc-d603-45e1-a958-e9d6b6ee4715)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715"
Nov 01 20:30:04.414994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.414906    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerStarted","Data":"06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"}
Nov 01 20:30:04.416348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.416326    5041 scope.go:122] "RemoveContainer" containerID="f6ccffdaba922e7fa641a6cd8df68a40722aad49d6f992b76db09607a7a62f44"
Nov 01 20:30:04.423353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.423329    5041 generic.go:356] "Generic (PLEG): container finished" podID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerID="db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa" exitCode=60
Nov 01 20:30:04.423703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.423684    5041 scope.go:122] "RemoveContainer" containerID="db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"
Nov 01 20:30:04.423965 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:04.423943    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-fsrpf_odh-ai-gateway-infra(c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb"
Nov 01 20:30:04.429981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.429952    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerStarted","Data":"44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"}
Nov 01 20:30:04.430105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.430008    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerDied","Data":"db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"}
Nov 01 20:30:04.449349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.449314    5041 scope.go:122] "RemoveContainer" containerID="23af230fb3fa3684c8f8ad674e78de8a4251b2ddbe53c46317c6624e229fddaf"
Nov 01 20:30:04.506949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.506925    5041 scope.go:122] "RemoveContainer" containerID="53be26e925414ab2dd5ed42968d208507174c780d76c0b89095954cc1562647b"
Nov 01 20:30:04.524045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.524023    5041 scope.go:122] "RemoveContainer" containerID="92666a7f2501837cd84b3dc0ecab8b870e6bfebac601d60a349bb00c4f05c6bf"
Nov 01 20:30:04.560837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:04.550361    5041 scope.go:122] "RemoveContainer" containerID="edd5ce69cb4a294b6facfcfa6521d8b07be83610910f3c30988df70fce446005"
Nov 01 20:30:05.436841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.436801    5041 generic.go:356] "Generic (PLEG): container finished" podID="280a0cea-2aee-436d-993e-1e7d302da192" containerID="1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6" exitCode=60
Nov 01 20:30:05.437351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.436882    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerDied","Data":"1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"}
Nov 01 20:30:05.437351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.436941    5041 scope.go:122] "RemoveContainer" containerID="9eac58665373858a6e8ed303314420ca41daceda3bb066c4ee610d5d8730e3d6"
Nov 01 20:30:05.437351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.437174    5041 scope.go:122] "RemoveContainer" containerID="1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"
Nov 01 20:30:05.437556 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.437528    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s_odh-ai-gateway-infra(280a0cea-2aee-436d-993e-1e7d302da192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" podUID="280a0cea-2aee-436d-993e-1e7d302da192"
Nov 01 20:30:05.439232 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.439211    5041 scope.go:122] "RemoveContainer" containerID="c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"
Nov 01 20:30:05.439505 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.439486    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl_odh-ai-gateway-infra(8aaf972e-adcb-492c-b944-0a17205f31e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2"
Nov 01 20:30:05.440730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.440711    5041 scope.go:122] "RemoveContainer" containerID="7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e"
Nov 01 20:30:05.441013 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.440960    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d_odh-ai-gateway-infra(26a780fc-d603-45e1-a958-e9d6b6ee4715)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715"
Nov 01 20:30:05.442525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.442503    5041 generic.go:356] "Generic (PLEG): container finished" podID="318364b7-2ac9-4888-a2ac-599131e38f74" containerID="06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa" exitCode=60
Nov 01 20:30:05.442636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.442568    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerDied","Data":"06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"}
Nov 01 20:30:05.442758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.442740    5041 scope.go:122] "RemoveContainer" containerID="06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"
Nov 01 20:30:05.443068 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.443022    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5_odh-ai-gateway-infra(318364b7-2ac9-4888-a2ac-599131e38f74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" podUID="318364b7-2ac9-4888-a2ac-599131e38f74"
Nov 01 20:30:05.444650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.444628    5041 generic.go:356] "Generic (PLEG): container finished" podID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerID="44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e" exitCode=60
Nov 01 20:30:05.444726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.444703    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerDied","Data":"44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"}
Nov 01 20:30:05.444938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.444920    5041 scope.go:122] "RemoveContainer" containerID="44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"
Nov 01 20:30:05.445152 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.445129    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8_odh-ai-gateway-infra(279d7000-3813-4aff-bff0-f2d0753a00ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee"
Nov 01 20:30:05.446666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.446646    5041 scope.go:122] "RemoveContainer" containerID="db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"
Nov 01 20:30:05.447532 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.446938    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-fsrpf_odh-ai-gateway-infra(c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb"
Nov 01 20:30:05.450192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.450170    5041 scope.go:122] "RemoveContainer" containerID="45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"
Nov 01 20:30:05.450429 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.450402    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf_odh-ai-gateway-infra(1fc3d458-db41-4e30-8cd3-c8280fb488f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0"
Nov 01 20:30:05.452635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.452615    5041 scope.go:122] "RemoveContainer" containerID="a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"
Nov 01 20:30:05.452904 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:05.452882    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm_odh-ai-gateway-infra(caef7d46-7470-41bf-897c-b181d3f94ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0"
Nov 01 20:30:05.453836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.453819    5041 scope.go:122] "RemoveContainer" containerID="53214c627825eb88da8ca9d525029cec4f2d4a08eca55542562bdfe52e8d0a26"
Nov 01 20:30:05.469044 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:05.469016    5041 scope.go:122] "RemoveContainer" containerID="1e89759fab8263942feb447e41a456196b510f5afdecea2106b11917522a51ab"
Nov 01 20:30:06.462980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:06.462940    5041 scope.go:122] "RemoveContainer" containerID="1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"
Nov 01 20:30:06.463452 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:06.463264    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s_odh-ai-gateway-infra(280a0cea-2aee-436d-993e-1e7d302da192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" podUID="280a0cea-2aee-436d-993e-1e7d302da192"
Nov 01 20:30:06.465055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:06.465031    5041 scope.go:122] "RemoveContainer" containerID="06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"
Nov 01 20:30:06.465353 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:06.465329    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5_odh-ai-gateway-infra(318364b7-2ac9-4888-a2ac-599131e38f74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" podUID="318364b7-2ac9-4888-a2ac-599131e38f74"
Nov 01 20:30:06.466809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:06.466757    5041 scope.go:122] "RemoveContainer" containerID="44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"
Nov 01 20:30:06.467010 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:06.466992    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8_odh-ai-gateway-infra(279d7000-3813-4aff-bff0-f2d0753a00ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee"
Nov 01 20:30:12.024511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:12.024475    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"]
Nov 01 20:30:12.030244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:12.030203    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb9c6ec-dq6bf"]
Nov 01 20:30:12.432212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:12.432179    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc737061-a9d4-4d3f-8172-dd05a993fc9b" path="/var/lib/kubelet/pods/bc737061-a9d4-4d3f-8172-dd05a993fc9b/volumes"
Nov 01 20:30:14.421280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:14.421246    5041 scope.go:122] "RemoveContainer" containerID="c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"
Nov 01 20:30:15.420889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.420863    5041 scope.go:122] "RemoveContainer" containerID="44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"
Nov 01 20:30:15.421076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.421034    5041 scope.go:122] "RemoveContainer" containerID="45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"
Nov 01 20:30:15.421200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.421177    5041 scope.go:122] "RemoveContainer" containerID="a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"
Nov 01 20:30:15.540047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.540016    5041 generic.go:356] "Generic (PLEG): container finished" podID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerID="c0e257875f03f3a0b674074996084e379bd6eb454537e5ac4594c3fdc05801ae" exitCode=60
Nov 01 20:30:15.540520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.540096    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerDied","Data":"c0e257875f03f3a0b674074996084e379bd6eb454537e5ac4594c3fdc05801ae"}
Nov 01 20:30:15.540520 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.540169    5041 scope.go:122] "RemoveContainer" containerID="c59f5073b592ec6c693aae56b3daf43e79d3aa56364c292001f799bbe8716a96"
Nov 01 20:30:15.540632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:15.540553    5041 scope.go:122] "RemoveContainer" containerID="c0e257875f03f3a0b674074996084e379bd6eb454537e5ac4594c3fdc05801ae"
Nov 01 20:30:15.540937 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:15.540916    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl_odh-ai-gateway-infra(8aaf972e-adcb-492c-b944-0a17205f31e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2"
Nov 01 20:30:16.551750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.551718    5041 generic.go:356] "Generic (PLEG): container finished" podID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerID="95dde13dd0a30065165eadcf48f11c6db83176edaf56db2336d9c7a8a23424e8" exitCode=60
Nov 01 20:30:16.552164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.551817    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerDied","Data":"95dde13dd0a30065165eadcf48f11c6db83176edaf56db2336d9c7a8a23424e8"}
Nov 01 20:30:16.552164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.551874    5041 scope.go:122] "RemoveContainer" containerID="44beff76f9a0fa7632f9f5d02010e973bdebb092c272e7bd4df123f025bc972e"
Nov 01 20:30:16.552292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.552271    5041 scope.go:122] "RemoveContainer" containerID="95dde13dd0a30065165eadcf48f11c6db83176edaf56db2336d9c7a8a23424e8"
Nov 01 20:30:16.552631 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:16.552605    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8_odh-ai-gateway-infra(279d7000-3813-4aff-bff0-f2d0753a00ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee"
Nov 01 20:30:16.554191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.554173    5041 generic.go:356] "Generic (PLEG): container finished" podID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerID="87ccc607f9454da5705dc42fa82824b1ab76e6ce464c4362823fe6061783c5ee" exitCode=60
Nov 01 20:30:16.554313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.554252    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerDied","Data":"87ccc607f9454da5705dc42fa82824b1ab76e6ce464c4362823fe6061783c5ee"}
Nov 01 20:30:16.554637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.554620    5041 scope.go:122] "RemoveContainer" containerID="87ccc607f9454da5705dc42fa82824b1ab76e6ce464c4362823fe6061783c5ee"
Nov 01 20:30:16.554987 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:16.554964    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf_odh-ai-gateway-infra(1fc3d458-db41-4e30-8cd3-c8280fb488f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0"
Nov 01 20:30:16.556346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.556329    5041 generic.go:356] "Generic (PLEG): container finished" podID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerID="5f6ee83b48040c3a0035ef5afbd3c5ec479eef7d4204af3d04710c611d26b8ad" exitCode=60
Nov 01 20:30:16.556461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.556352    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerDied","Data":"5f6ee83b48040c3a0035ef5afbd3c5ec479eef7d4204af3d04710c611d26b8ad"}
Nov 01 20:30:16.556668 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.556654    5041 scope.go:122] "RemoveContainer" containerID="5f6ee83b48040c3a0035ef5afbd3c5ec479eef7d4204af3d04710c611d26b8ad"
Nov 01 20:30:16.556953 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:16.556933    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm_odh-ai-gateway-infra(caef7d46-7470-41bf-897c-b181d3f94ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0"
Nov 01 20:30:16.567686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.567652    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:16.588122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.588096    5041 scope.go:122] "RemoveContainer" containerID="45b5f5f6646bb8e3df5a5bfd568758df3ebd2e6ffa29cf2708cd9821dbf6d8b5"
Nov 01 20:30:16.605479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.605453    5041 scope.go:122] "RemoveContainer" containerID="a0ab319a4dcc0e64ceb72a236e1f44fbccc0ed3148b348b1a6af78220c91e009"
Nov 01 20:30:16.732697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.732669    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:16.794442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.794392    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\" (UniqueName: \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\") pod \"8aaf972e-adcb-492c-b944-0a17205f31e2\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") "
Nov 01 20:30:16.794716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.794693    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\") pod \"8aaf972e-adcb-492c-b944-0a17205f31e2\" (UID: \"8aaf972e-adcb-492c-b944-0a17205f31e2\") "
Nov 01 20:30:16.795453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.795136    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca" pod "8aaf972e-adcb-492c-b944-0a17205f31e2" (UID: "8aaf972e-adcb-492c-b944-0a17205f31e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:16.795453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.795414    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aaf972e-adcb-492c-b944-0a17205f31e2-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:16.797229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.797198    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5" pod "8aaf972e-adcb-492c-b944-0a17205f31e2" (UID: "8aaf972e-adcb-492c-b944-0a17205f31e2"). InnerVolumeSpecName "kube-api-access-pgks5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:16.896886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:16.896801    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgks5\" (UniqueName: \"kubernetes.io/projected/8aaf972e-adcb-492c-b944-0a17205f31e2-kube-api-access-pgks5\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:17.420995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.420962    5041 scope.go:122] "RemoveContainer" containerID="7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e"
Nov 01 20:30:17.421410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.421351    5041 scope.go:122] "RemoveContainer" containerID="e4eee5b1841128b33b21324e6283274bbabd5f227b969d5ad66973583048cdcb"
Nov 01 20:30:17.421581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.421491    5041 scope.go:122] "RemoveContainer" containerID="db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"
Nov 01 20:30:17.421823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.421764    5041 scope.go:122] "RemoveContainer" containerID="1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"
Nov 01 20:30:17.570930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.570890    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl" event={"ID":"8aaf972e-adcb-492c-b944-0a17205f31e2","Type":"ContainerDied","Data":"c9a06d8ee3d6a01b993c1062afb4fd5cc23a05836c720a8457661b0ca14a6487"}
Nov 01 20:30:17.570930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.570922    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"
Nov 01 20:30:17.571418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.570966    5041 scope.go:122] "RemoveContainer" containerID="c0e257875f03f3a0b674074996084e379bd6eb454537e5ac4594c3fdc05801ae"
Nov 01 20:30:17.585386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.585338    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:17.595305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.595281    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:17.615747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.615720    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:17.631304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.631265    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:17.640443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.640416    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0aa61b-29892750-ccrjl"]
Nov 01 20:30:17.847751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.847721    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:17.960079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.960047    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:17.967577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:17.967547    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:18.025093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.025053    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\" (UniqueName: \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\") pod \"caef7d46-7470-41bf-897c-b181d3f94ff0\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") "
Nov 01 20:30:18.025305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.025174    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\") pod \"caef7d46-7470-41bf-897c-b181d3f94ff0\" (UID: \"caef7d46-7470-41bf-897c-b181d3f94ff0\") "
Nov 01 20:30:18.025565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.025539    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca" pod "caef7d46-7470-41bf-897c-b181d3f94ff0" (UID: "caef7d46-7470-41bf-897c-b181d3f94ff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:18.027080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.025820    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caef7d46-7470-41bf-897c-b181d3f94ff0-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.027673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.027649    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2" pod "caef7d46-7470-41bf-897c-b181d3f94ff0" (UID: "caef7d46-7470-41bf-897c-b181d3f94ff0"). InnerVolumeSpecName "kube-api-access-kwpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:18.126721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.126690    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\" (UniqueName: \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\") pod \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") "
Nov 01 20:30:18.126952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.126813    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\") pod \"279d7000-3813-4aff-bff0-f2d0753a00ee\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") "
Nov 01 20:30:18.126952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.126909    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\") pod \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\" (UID: \"1fc3d458-db41-4e30-8cd3-c8280fb488f0\") "
Nov 01 20:30:18.127149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127128    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\" (UniqueName: \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\") pod \"279d7000-3813-4aff-bff0-f2d0753a00ee\" (UID: \"279d7000-3813-4aff-bff0-f2d0753a00ee\") "
Nov 01 20:30:18.127233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127195    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca" pod "279d7000-3813-4aff-bff0-f2d0753a00ee" (UID: "279d7000-3813-4aff-bff0-f2d0753a00ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:18.127313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127276    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca" pod "1fc3d458-db41-4e30-8cd3-c8280fb488f0" (UID: "1fc3d458-db41-4e30-8cd3-c8280fb488f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:18.127643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127625    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/279d7000-3813-4aff-bff0-f2d0753a00ee-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.127716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127648    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fc3d458-db41-4e30-8cd3-c8280fb488f0-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.127716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.127664    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwpt2\" (UniqueName: \"kubernetes.io/projected/caef7d46-7470-41bf-897c-b181d3f94ff0-kube-api-access-kwpt2\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.128935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.128916    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg" pod "1fc3d458-db41-4e30-8cd3-c8280fb488f0" (UID: "1fc3d458-db41-4e30-8cd3-c8280fb488f0"). InnerVolumeSpecName "kube-api-access-kwprg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:18.129065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.129042    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88" pod "279d7000-3813-4aff-bff0-f2d0753a00ee" (UID: "279d7000-3813-4aff-bff0-f2d0753a00ee"). InnerVolumeSpecName "kube-api-access-fbv88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:18.229327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.229207    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwprg\" (UniqueName: \"kubernetes.io/projected/1fc3d458-db41-4e30-8cd3-c8280fb488f0-kube-api-access-kwprg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.229327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.229248    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbv88\" (UniqueName: \"kubernetes.io/projected/279d7000-3813-4aff-bff0-f2d0753a00ee-kube-api-access-fbv88\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:18.421610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.421569    5041 scope.go:122] "RemoveContainer" containerID="06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"
Nov 01 20:30:18.434485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.434451    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" path="/var/lib/kubelet/pods/8aaf972e-adcb-492c-b944-0a17205f31e2/volumes"
Nov 01 20:30:18.580754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.580716    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8" event={"ID":"279d7000-3813-4aff-bff0-f2d0753a00ee","Type":"ContainerDied","Data":"96bb6266b900cf344fa45e2d918358e6004cac4bcd70b5b34c9c60d05c2ee184"}
Nov 01 20:30:18.581270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.580726    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"
Nov 01 20:30:18.581270 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.580802    5041 scope.go:122] "RemoveContainer" containerID="95dde13dd0a30065165eadcf48f11c6db83176edaf56db2336d9c7a8a23424e8"
Nov 01 20:30:18.583407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.583381    5041 generic.go:356] "Generic (PLEG): container finished" podID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerID="a619163b0c00ca55037c5951ad69b0dcc99b00a37212176b68f0c54be36ae8b4" exitCode=60
Nov 01 20:30:18.583538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.583463    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerDied","Data":"a619163b0c00ca55037c5951ad69b0dcc99b00a37212176b68f0c54be36ae8b4"}
Nov 01 20:30:18.584077 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.584046    5041 scope.go:122] "RemoveContainer" containerID="a619163b0c00ca55037c5951ad69b0dcc99b00a37212176b68f0c54be36ae8b4"
Nov 01 20:30:18.584428 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:18.584403    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892750-fsrpf_odh-ai-gateway-infra(c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb"
Nov 01 20:30:18.585995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.585974    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"
Nov 01 20:30:18.586110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.586015    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf" event={"ID":"1fc3d458-db41-4e30-8cd3-c8280fb488f0","Type":"ContainerDied","Data":"c00155003d28a8c76aa420b3642d4d2c900f4bdb65d5a2ab9c190b5e8cc13716"}
Nov 01 20:30:18.587631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.587601    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"
Nov 01 20:30:18.587631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.587621    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm" event={"ID":"caef7d46-7470-41bf-897c-b181d3f94ff0","Type":"ContainerDied","Data":"86bfd58ae102623e32f3e668fb7d6fc172c409eb927782c5512f5f859a78f898"}
Nov 01 20:30:18.590292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.590271    5041 generic.go:356] "Generic (PLEG): container finished" podID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerID="71968e934a7e0127cdd350e187ac3cae1e180e5148dbef9d5cfc6e56836b1874" exitCode=60
Nov 01 20:30:18.590397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.590342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" event={"ID":"fa101d42-7dc9-4f51-9997-77166e0bcc98","Type":"ContainerDied","Data":"71968e934a7e0127cdd350e187ac3cae1e180e5148dbef9d5cfc6e56836b1874"}
Nov 01 20:30:18.590774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.590750    5041 scope.go:122] "RemoveContainer" containerID="71968e934a7e0127cdd350e187ac3cae1e180e5148dbef9d5cfc6e56836b1874"
Nov 01 20:30:18.591308 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:18.591049    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx_odh-ai-gateway-infra(fa101d42-7dc9-4f51-9997-77166e0bcc98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98"
Nov 01 20:30:18.592739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.592715    5041 generic.go:356] "Generic (PLEG): container finished" podID="280a0cea-2aee-436d-993e-1e7d302da192" containerID="48bfde73005e277764e7c8301de2126b026faa81583092526b61a09afe50d0d4" exitCode=60
Nov 01 20:30:18.592865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.592819    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerDied","Data":"48bfde73005e277764e7c8301de2126b026faa81583092526b61a09afe50d0d4"}
Nov 01 20:30:18.593218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.593196    5041 scope.go:122] "RemoveContainer" containerID="48bfde73005e277764e7c8301de2126b026faa81583092526b61a09afe50d0d4"
Nov 01 20:30:18.593482 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:18.593454    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s_odh-ai-gateway-infra(280a0cea-2aee-436d-993e-1e7d302da192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" podUID="280a0cea-2aee-436d-993e-1e7d302da192"
Nov 01 20:30:18.596213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.596169    5041 generic.go:356] "Generic (PLEG): container finished" podID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerID="31cdac229680eb74b9f524d0fcc32535286136c68ec602af1f8ff6f52a22e55b" exitCode=60
Nov 01 20:30:18.596213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.596208    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" event={"ID":"26a780fc-d603-45e1-a958-e9d6b6ee4715","Type":"ContainerDied","Data":"31cdac229680eb74b9f524d0fcc32535286136c68ec602af1f8ff6f52a22e55b"}
Nov 01 20:30:18.596567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.596549    5041 scope.go:122] "RemoveContainer" containerID="31cdac229680eb74b9f524d0fcc32535286136c68ec602af1f8ff6f52a22e55b"
Nov 01 20:30:18.596872 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:18.596830    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d_odh-ai-gateway-infra(26a780fc-d603-45e1-a958-e9d6b6ee4715)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715"
Nov 01 20:30:18.695834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.695610    5041 scope.go:122] "RemoveContainer" containerID="db2a41b5596712903ed99d7b90b2f4a1cfb7b232a3d629efd0da22a891ec08fa"
Nov 01 20:30:18.720872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.720851    5041 scope.go:122] "RemoveContainer" containerID="87ccc607f9454da5705dc42fa82824b1ab76e6ce464c4362823fe6061783c5ee"
Nov 01 20:30:18.730974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.730951    5041 scope.go:122] "RemoveContainer" containerID="5f6ee83b48040c3a0035ef5afbd3c5ec479eef7d4204af3d04710c611d26b8ad"
Nov 01 20:30:18.738056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.737994    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:18.742912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.742886    5041 scope.go:122] "RemoveContainer" containerID="e4eee5b1841128b33b21324e6283274bbabd5f227b969d5ad66973583048cdcb"
Nov 01 20:30:18.750135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.750110    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bccce1-29892750-wvgt8"]
Nov 01 20:30:18.758935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.758911    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:18.763249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.763227    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-9be196-29892750-vk2kf"]
Nov 01 20:30:18.770226 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.770201    5041 scope.go:122] "RemoveContainer" containerID="1c127425de84c150b8abc1ea35555ed13a2d2705096500ade2fe3f423c7be8b6"
Nov 01 20:30:18.774494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.774472    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:18.778382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.778357    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-036c69-29892750-lr6bm"]
Nov 01 20:30:18.785488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:18.785473    5041 scope.go:122] "RemoveContainer" containerID="7ffe18a474fe13f203975c8cc08bfc7f49c7ffa44722934c4911bda640ae940e"
Nov 01 20:30:19.605870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.605835    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerStarted","Data":"3caf633c548bfba9a0d750ea61abb79af472d36ce157ccf6a18a8e17e496bf8a"}
Nov 01 20:30:19.613184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.613155    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:19.648400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.648362    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:19.675559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.675527    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:19.706477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.706049    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:19.865983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.865657    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:19.949307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.949276    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:19.963690 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.963656    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\") pod \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") "
Nov 01 20:30:19.963962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.963848    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\" (UniqueName: \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\") pod \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\" (UID: \"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb\") "
Nov 01 20:30:19.965090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.965065    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca" pod "c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" (UID: "c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:19.968715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.968691    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv" pod "c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" (UID: "c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb"). InnerVolumeSpecName "kube-api-access-x6wwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:19.987106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:19.987078    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:20.017083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.017057    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:20.064862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.064824    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\") pod \"280a0cea-2aee-436d-993e-1e7d302da192\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") "
Nov 01 20:30:20.065186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.065164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\" (UniqueName: \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\") pod \"280a0cea-2aee-436d-993e-1e7d302da192\" (UID: \"280a0cea-2aee-436d-993e-1e7d302da192\") "
Nov 01 20:30:20.065294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.065236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca" pod "280a0cea-2aee-436d-993e-1e7d302da192" (UID: "280a0cea-2aee-436d-993e-1e7d302da192"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:20.065705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.065681    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.065705 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.065707    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6wwv\" (UniqueName: \"kubernetes.io/projected/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb-kube-api-access-x6wwv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.065929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.065729    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280a0cea-2aee-436d-993e-1e7d302da192-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.067437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.067407    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv" pod "280a0cea-2aee-436d-993e-1e7d302da192" (UID: "280a0cea-2aee-436d-993e-1e7d302da192"). InnerVolumeSpecName "kube-api-access-sqtnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:20.166290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\") pod \"fa101d42-7dc9-4f51-9997-77166e0bcc98\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") "
Nov 01 20:30:20.166290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166246    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\") pod \"26a780fc-d603-45e1-a958-e9d6b6ee4715\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") "
Nov 01 20:30:20.166290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166290    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\" (UniqueName: \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\") pod \"26a780fc-d603-45e1-a958-e9d6b6ee4715\" (UID: \"26a780fc-d603-45e1-a958-e9d6b6ee4715\") "
Nov 01 20:30:20.166593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166353    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\" (UniqueName: \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\") pod \"fa101d42-7dc9-4f51-9997-77166e0bcc98\" (UID: \"fa101d42-7dc9-4f51-9997-77166e0bcc98\") "
Nov 01 20:30:20.166660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166637    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca" pod "fa101d42-7dc9-4f51-9997-77166e0bcc98" (UID: "fa101d42-7dc9-4f51-9997-77166e0bcc98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:20.166713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166652    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca" pod "26a780fc-d603-45e1-a958-e9d6b6ee4715" (UID: "26a780fc-d603-45e1-a958-e9d6b6ee4715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:20.166814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166787    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa101d42-7dc9-4f51-9997-77166e0bcc98-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.166878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166817    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26a780fc-d603-45e1-a958-e9d6b6ee4715-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.166878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.166836    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtnv\" (UniqueName: \"kubernetes.io/projected/280a0cea-2aee-436d-993e-1e7d302da192-kube-api-access-sqtnv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.168598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.168576    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l" pod "fa101d42-7dc9-4f51-9997-77166e0bcc98" (UID: "fa101d42-7dc9-4f51-9997-77166e0bcc98"). InnerVolumeSpecName "kube-api-access-h2b2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:20.168703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.168660    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t" pod "26a780fc-d603-45e1-a958-e9d6b6ee4715" (UID: "26a780fc-d603-45e1-a958-e9d6b6ee4715"). InnerVolumeSpecName "kube-api-access-tsd7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:20.268568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.268527    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsd7t\" (UniqueName: \"kubernetes.io/projected/26a780fc-d603-45e1-a958-e9d6b6ee4715-kube-api-access-tsd7t\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.268568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.268563    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2b2l\" (UniqueName: \"kubernetes.io/projected/fa101d42-7dc9-4f51-9997-77166e0bcc98-kube-api-access-h2b2l\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.432068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.431989    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" path="/var/lib/kubelet/pods/1fc3d458-db41-4e30-8cd3-c8280fb488f0/volumes"
Nov 01 20:30:20.432482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.432467    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" path="/var/lib/kubelet/pods/279d7000-3813-4aff-bff0-f2d0753a00ee/volumes"
Nov 01 20:30:20.432893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.432879    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" path="/var/lib/kubelet/pods/caef7d46-7470-41bf-897c-b181d3f94ff0/volumes"
Nov 01 20:30:20.626615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.626583    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"
Nov 01 20:30:20.626615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.626593    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf" event={"ID":"c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb","Type":"ContainerDied","Data":"785551fbf9464f6215fc72d5602d1b9a83a7b4ae253fdc8045b11c119201903b"}
Nov 01 20:30:20.627370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.626678    5041 scope.go:122] "RemoveContainer" containerID="a619163b0c00ca55037c5951ad69b0dcc99b00a37212176b68f0c54be36ae8b4"
Nov 01 20:30:20.629617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.629596    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"
Nov 01 20:30:20.629718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.629593    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx" event={"ID":"fa101d42-7dc9-4f51-9997-77166e0bcc98","Type":"ContainerDied","Data":"5c667cbcc27406403aaa8cd1b516e5102289a469fcf9345d729a83f8a4cebd83"}
Nov 01 20:30:20.631352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.631301    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"
Nov 01 20:30:20.631352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.631338    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s" event={"ID":"280a0cea-2aee-436d-993e-1e7d302da192","Type":"ContainerDied","Data":"0fd89898dfe6638243ff42f64cb9a4803160e013d1f24255b5fc642cc236b346"}
Nov 01 20:30:20.633058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.633038    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"
Nov 01 20:30:20.633157 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.633057    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d" event={"ID":"26a780fc-d603-45e1-a958-e9d6b6ee4715","Type":"ContainerDied","Data":"f6839bf4c18d20d37f5e62544c98b663a49e7cddd672844612dfa0bea8f74bbd"}
Nov 01 20:30:20.634605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.634582    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:20.635472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.635447    5041 generic.go:356] "Generic (PLEG): container finished" podID="318364b7-2ac9-4888-a2ac-599131e38f74" containerID="3caf633c548bfba9a0d750ea61abb79af472d36ce157ccf6a18a8e17e496bf8a" exitCode=60
Nov 01 20:30:20.635546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.635502    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerDied","Data":"3caf633c548bfba9a0d750ea61abb79af472d36ce157ccf6a18a8e17e496bf8a"}
Nov 01 20:30:20.641341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.641321    5041 scope.go:122] "RemoveContainer" containerID="71968e934a7e0127cdd350e187ac3cae1e180e5148dbef9d5cfc6e56836b1874"
Nov 01 20:30:20.647024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.647001    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:20.654245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.654204    5041 scope.go:122] "RemoveContainer" containerID="48bfde73005e277764e7c8301de2126b026faa81583092526b61a09afe50d0d4"
Nov 01 20:30:20.659450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.659426    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892750-fsrpf"]
Nov 01 20:30:20.665238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.665213    5041 scope.go:122] "RemoveContainer" containerID="31cdac229680eb74b9f524d0fcc32535286136c68ec602af1f8ff6f52a22e55b"
Nov 01 20:30:20.677872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.677847    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:20.678893 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.678808    5041 scope.go:122] "RemoveContainer" containerID="06886a6db25d9c2a06581b00095f654d6adb4a109a2edef53091f8e7fd49cfaa"
Nov 01 20:30:20.683290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.683245    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892750-cwf8s"]
Nov 01 20:30:20.693724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.693701    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:20.701198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.701178    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8b6cd6-29892750-cff5d"]
Nov 01 20:30:20.738907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.738869    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:20.780832 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.775576    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3a6480-29892750-fljbx"]
Nov 01 20:30:20.857611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.857581    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:20.879116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.879082    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\" (UniqueName: \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\") pod \"318364b7-2ac9-4888-a2ac-599131e38f74\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") "
Nov 01 20:30:20.879265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.879167    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\") pod \"318364b7-2ac9-4888-a2ac-599131e38f74\" (UID: \"318364b7-2ac9-4888-a2ac-599131e38f74\") "
Nov 01 20:30:20.880068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.880039    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca" pod "318364b7-2ac9-4888-a2ac-599131e38f74" (UID: "318364b7-2ac9-4888-a2ac-599131e38f74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:20.881861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.881829    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl" pod "318364b7-2ac9-4888-a2ac-599131e38f74" (UID: "318364b7-2ac9-4888-a2ac-599131e38f74"). InnerVolumeSpecName "kube-api-access-7gdcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:20.980960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.980871    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gdcl\" (UniqueName: \"kubernetes.io/projected/318364b7-2ac9-4888-a2ac-599131e38f74-kube-api-access-7gdcl\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:20.980960 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:20.980898    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318364b7-2ac9-4888-a2ac-599131e38f74-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:21.661534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:21.661496    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5" event={"ID":"318364b7-2ac9-4888-a2ac-599131e38f74","Type":"ContainerDied","Data":"16224d7f2ed1fd1e93988b87feb75e2c55245d9174473864440f93d3bcf742c0"}
Nov 01 20:30:21.662511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:21.662489    5041 scope.go:122] "RemoveContainer" containerID="3caf633c548bfba9a0d750ea61abb79af472d36ce157ccf6a18a8e17e496bf8a"
Nov 01 20:30:21.662962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:21.661961    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"
Nov 01 20:30:21.706804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:21.704087    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:21.724101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:21.721446    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e38ef4-29892750-tbtx5"]
Nov 01 20:30:22.436980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:22.436945    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" path="/var/lib/kubelet/pods/26a780fc-d603-45e1-a958-e9d6b6ee4715/volumes"
Nov 01 20:30:22.437736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:22.437660    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280a0cea-2aee-436d-993e-1e7d302da192" path="/var/lib/kubelet/pods/280a0cea-2aee-436d-993e-1e7d302da192/volumes"
Nov 01 20:30:22.438342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:22.438323    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" path="/var/lib/kubelet/pods/318364b7-2ac9-4888-a2ac-599131e38f74/volumes"
Nov 01 20:30:22.439400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:22.439372    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" path="/var/lib/kubelet/pods/c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb/volumes"
Nov 01 20:30:22.439904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:22.439887    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" path="/var/lib/kubelet/pods/fa101d42-7dc9-4f51-9997-77166e0bcc98/volumes"
Nov 01 20:30:28.542072 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:28.542034    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:30:29.133017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.132981    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"]
Nov 01 20:30:29.133570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133557    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133572    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133582    5041 cpu_manager.go:414] "containerMap: removing container" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133587    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133598    5041 cpu_manager.go:414] "containerMap: removing container" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133603    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.133616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133614    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133619    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133632    5041 cpu_manager.go:414] "containerMap: removing container" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133638    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133645    5041 cpu_manager.go:414] "containerMap: removing container" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133649    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133657    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133662    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133668    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133672    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133677    5041 cpu_manager.go:414] "containerMap: removing container" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133681    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133692    5041 cpu_manager.go:414] "containerMap: removing container" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133696    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133704    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133709    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133718    5041 cpu_manager.go:414] "containerMap: removing container" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133722    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133729    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133733    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133744    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133748    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133754    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133758    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133765    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133769    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133774    5041 cpu_manager.go:414] "containerMap: removing container" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133791    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133798    5041 cpu_manager.go:414] "containerMap: removing container" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133802    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133809    5041 cpu_manager.go:414] "containerMap: removing container" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133813    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133822    5041 cpu_manager.go:414] "containerMap: removing container" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133826    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133834    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133838    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133844    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133849    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133860    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133864    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133871    5041 cpu_manager.go:414] "containerMap: removing container" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133875    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.133937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133954    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133961    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133968    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133973    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133981    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133987    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133992    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.133998    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134005    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134012    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134016    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134021    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134026    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134031    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134037    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134042    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134047    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="318364b7-2ac9-4888-a2ac-599131e38f74" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134054    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134059    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:29.135304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.134064    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:29.137603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.137586    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.141749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.141724    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:30:29.142849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.142593    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:30:29.142849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.142668    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:30:29.142849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.142818    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:30:29.147100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.147060    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"]
Nov 01 20:30:29.216762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.216725    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.216989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.216951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.217038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.217011    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6b9\" (UniqueName: \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.317950 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.317916    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.318108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.317992    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6b9\" (UniqueName: \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.318186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.318132    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.318806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.318757    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.320443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.320417    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.325910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.325884    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6b9\" (UniqueName: \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\") pod \"maas-api-revoke-sub-76900056-z72vg\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.453818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.453722    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:29.733922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:29.733895    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"]
Nov 01 20:30:29.735542 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:29.735505    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0b3905_ccc0_433d_8bf8_61db087fef5d.slice/crio-b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad WatchSource:0}: Error finding container b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad: Status 404 returned error can't find the container with id b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad
Nov 01 20:30:30.745386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:30.745346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg" event={"ID":"cb0b3905-ccc0-433d-8bf8-61db087fef5d","Type":"ContainerStarted","Data":"37bcd88c88a959aad366dded7ed244f6eee228cb2ab63f85b0750de526ea2c72"}
Nov 01 20:30:30.745866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:30.745403    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg" event={"ID":"cb0b3905-ccc0-433d-8bf8-61db087fef5d","Type":"ContainerStarted","Data":"b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad"}
Nov 01 20:30:30.762009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:30.761962    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg" podStartSLOduration=1.7619473970000001 podStartE2EDuration="1.761947397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:30.760229416 +0000 UTC m=+1918.481308092" watchObservedRunningTime="2026-11-01 20:30:30.761947397 +0000 UTC m=+1918.483026072"
Nov 01 20:30:31.756127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:31.756095    5041 generic.go:356] "Generic (PLEG): container finished" podID="cb0b3905-ccc0-433d-8bf8-61db087fef5d" containerID="37bcd88c88a959aad366dded7ed244f6eee228cb2ab63f85b0750de526ea2c72" exitCode=0
Nov 01 20:30:31.756571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:31.756172    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg" event={"ID":"cb0b3905-ccc0-433d-8bf8-61db087fef5d","Type":"ContainerDied","Data":"37bcd88c88a959aad366dded7ed244f6eee228cb2ab63f85b0750de526ea2c72"}
Nov 01 20:30:32.929298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:32.929275    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:33.071825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.071773    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\") pod \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") "
Nov 01 20:30:33.072005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.071975    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\" (UniqueName: \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\") pod \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") "
Nov 01 20:30:33.072057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.072039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\") pod \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\" (UID: \"cb0b3905-ccc0-433d-8bf8-61db087fef5d\") "
Nov 01 20:30:33.072349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.072325    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca" pod "cb0b3905-ccc0-433d-8bf8-61db087fef5d" (UID: "cb0b3905-ccc0-433d-8bf8-61db087fef5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:33.072566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.072549    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:33.073990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.073968    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca" pod "cb0b3905-ccc0-433d-8bf8-61db087fef5d" (UID: "cb0b3905-ccc0-433d-8bf8-61db087fef5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:30:33.074101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.074033    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9" pod "cb0b3905-ccc0-433d-8bf8-61db087fef5d" (UID: "cb0b3905-ccc0-433d-8bf8-61db087fef5d"). InnerVolumeSpecName "kube-api-access-pt6b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:33.173578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.173544    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6b9\" (UniqueName: \"kubernetes.io/projected/cb0b3905-ccc0-433d-8bf8-61db087fef5d-kube-api-access-pt6b9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:33.173578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.173574    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0b3905-ccc0-433d-8bf8-61db087fef5d-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:33.774418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.774385    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"
Nov 01 20:30:33.774577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.774428    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg" event={"ID":"cb0b3905-ccc0-433d-8bf8-61db087fef5d","Type":"ContainerDied","Data":"b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad"}
Nov 01 20:30:33.774577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:33.774476    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02382d14877e95aa2ba4dc0e10e058a9044389d0d6ba31cb27d0dc4076a61ad"
Nov 01 20:30:36.099563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:36.099530    5041 scope.go:122] "RemoveContainer" containerID="f2ddbafd6bbbf8e5267c7c7a42ec49fc719bb9c3c80dea80c6e0dfcd13154e09"
Nov 01 20:30:36.112360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:36.112338    5041 scope.go:122] "RemoveContainer" containerID="2541f233694f57b9a6062b7349bf9ec51e4d83a187069b9f5adfc8dd7db25e61"
Nov 01 20:30:36.126745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:36.126721    5041 scope.go:122] "RemoveContainer" containerID="a7524d1935f49f2ae76f9ae2cbbe4369335cdcc7b378a28cc1d00daf396c5a52"
Nov 01 20:30:39.499508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.499466    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"]
Nov 01 20:30:39.501225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501191    5041 cpu_manager.go:414] "containerMap: removing container" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:39.501417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501401    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:39.501559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501546    5041 cpu_manager.go:414] "containerMap: removing container" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:39.501643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501632    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a0cea-2aee-436d-993e-1e7d302da192" containerName="cleanup"
Nov 01 20:30:39.501748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501736    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cb0b3905-ccc0-433d-8bf8-61db087fef5d" containerName="revoke-keys"
Nov 01 20:30:39.501875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501862    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0b3905-ccc0-433d-8bf8-61db087fef5d" containerName="revoke-keys"
Nov 01 20:30:39.501986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.501975    5041 cpu_manager.go:414] "containerMap: removing container" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:39.502075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.502064    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:39.502329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.502314    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa101d42-7dc9-4f51-9997-77166e0bcc98" containerName="cleanup"
Nov 01 20:30:39.502440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.502429    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0b3905-ccc0-433d-8bf8-61db087fef5d" containerName="revoke-keys"
Nov 01 20:30:39.502682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.502527    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a780fc-d603-45e1-a958-e9d6b6ee4715" containerName="cleanup"
Nov 01 20:30:39.502921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.502893    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="279d7000-3813-4aff-bff0-f2d0753a00ee" containerName="cleanup"
Nov 01 20:30:39.503098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.503085    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f7c3ee-5e67-419d-ba33-0d7221a2d1eb" containerName="cleanup"
Nov 01 20:30:39.503216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.503202    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fc3d458-db41-4e30-8cd3-c8280fb488f0" containerName="cleanup"
Nov 01 20:30:39.503315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.503304    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="caef7d46-7470-41bf-897c-b181d3f94ff0" containerName="cleanup"
Nov 01 20:30:39.503399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.503389    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf972e-adcb-492c-b944-0a17205f31e2" containerName="cleanup"
Nov 01 20:30:39.509976 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.509953    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.513170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.513145    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"]
Nov 01 20:30:39.519846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.517077    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:30:39.519846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.517223    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:30:39.519846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.517266    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:30:39.519846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.517584    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:30:39.616970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.616927    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.617160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.616983    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.617203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.617146    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ndgk\" (UniqueName: \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.719025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.718981    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.719181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.719037    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.719181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.719164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ndgk\" (UniqueName: \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.719771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.719741    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.721391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.721365    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.727275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.727247    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ndgk\" (UniqueName: \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\") pod \"maas-api-revoke-sub-e2e02845-p9xw4\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:39.829561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:39.829520    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:40.172866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.172833    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"]
Nov 01 20:30:40.175540 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:40.175504    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12392895_6c60_46a2_b646_09f9efe18b4a.slice/crio-60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a WatchSource:0}: Error finding container 60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a: Status 404 returned error can't find the container with id 60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a
Nov 01 20:30:40.577316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.577276    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:30:40.578675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.578260    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" containerID="cri-o://b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4" gracePeriod=30
Nov 01 20:30:40.615857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.614330    5041 patch_prober.go:29] interesting pod/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:30:40.615857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.614429    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:30:40.937678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.937587    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4" event={"ID":"12392895-6c60-46a2-b646-09f9efe18b4a","Type":"ContainerStarted","Data":"81f90c5b82060d8c80f660d69eefe0af5f6394d372b8c0695a08bd7a7a5954a6"}
Nov 01 20:30:40.937973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.937941    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4" event={"ID":"12392895-6c60-46a2-b646-09f9efe18b4a","Type":"ContainerStarted","Data":"60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a"}
Nov 01 20:30:40.989803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:40.989724    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4" podStartSLOduration=1.989710689 podStartE2EDuration="1.989710689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:40.989374698 +0000 UTC m=+1928.710453375" watchObservedRunningTime="2026-11-01 20:30:40.989710689 +0000 UTC m=+1928.710789365"
Nov 01 20:30:41.949234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:41.949199    5041 generic.go:356] "Generic (PLEG): container finished" podID="12392895-6c60-46a2-b646-09f9efe18b4a" containerID="81f90c5b82060d8c80f660d69eefe0af5f6394d372b8c0695a08bd7a7a5954a6" exitCode=0
Nov 01 20:30:41.949746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:41.949285    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4" event={"ID":"12392895-6c60-46a2-b646-09f9efe18b4a","Type":"ContainerDied","Data":"81f90c5b82060d8c80f660d69eefe0af5f6394d372b8c0695a08bd7a7a5954a6"}
Nov 01 20:30:43.158282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.158250    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:43.289591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.289555    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\") pod \"12392895-6c60-46a2-b646-09f9efe18b4a\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") "
Nov 01 20:30:43.289753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.289606    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\") pod \"12392895-6c60-46a2-b646-09f9efe18b4a\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") "
Nov 01 20:30:43.289856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.289759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\" (UniqueName: \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\") pod \"12392895-6c60-46a2-b646-09f9efe18b4a\" (UID: \"12392895-6c60-46a2-b646-09f9efe18b4a\") "
Nov 01 20:30:43.290022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.289989    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca" pod "12392895-6c60-46a2-b646-09f9efe18b4a" (UID: "12392895-6c60-46a2-b646-09f9efe18b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:43.290439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.290414    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:43.292057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.292030    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca" pod "12392895-6c60-46a2-b646-09f9efe18b4a" (UID: "12392895-6c60-46a2-b646-09f9efe18b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:30:43.292165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.292065    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk" pod "12392895-6c60-46a2-b646-09f9efe18b4a" (UID: "12392895-6c60-46a2-b646-09f9efe18b4a"). InnerVolumeSpecName "kube-api-access-7ndgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:43.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.392054    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ndgk\" (UniqueName: \"kubernetes.io/projected/12392895-6c60-46a2-b646-09f9efe18b4a-kube-api-access-7ndgk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:43.392094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.392090    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12392895-6c60-46a2-b646-09f9efe18b4a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:43.968479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.968444    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"
Nov 01 20:30:43.968673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.968464    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4" event={"ID":"12392895-6c60-46a2-b646-09f9efe18b4a","Type":"ContainerDied","Data":"60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a"}
Nov 01 20:30:43.968673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:43.968542    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60cc02afebf38d8a0a357feb4c74325411d881d45c9a37375406198fd452f17a"
Nov 01 20:30:44.018721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:44.018674    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:30:44.019033 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:44.018979    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" podUID="a4d0f35a-b93d-4245-8745-7f93722536cd" containerName="istio-proxy" containerID="cri-o://8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f" gracePeriod=30
Nov 01 20:30:45.928265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.928237    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:30:45.971648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.971623    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:30:45.987422 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.987393    5041 generic.go:356] "Generic (PLEG): container finished" podID="a4d0f35a-b93d-4245-8745-7f93722536cd" containerID="8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f" exitCode=0
Nov 01 20:30:45.987591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.987449    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"
Nov 01 20:30:45.987591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.987478    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" event={"ID":"a4d0f35a-b93d-4245-8745-7f93722536cd","Type":"ContainerDied","Data":"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"}
Nov 01 20:30:45.987591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.987536    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz" event={"ID":"a4d0f35a-b93d-4245-8745-7f93722536cd","Type":"ContainerDied","Data":"f1d8284f6fb0a83fb9fdcc36ce808f920b10ab10abd9e533f3e4893a2dc17ef7"}
Nov 01 20:30:45.987591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.987560    5041 scope.go:122] "RemoveContainer" containerID="8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"
Nov 01 20:30:45.989544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.989520    5041 generic.go:356] "Generic (PLEG): container finished" podID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerID="b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4" exitCode=0
Nov 01 20:30:45.989677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.989566    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"
Nov 01 20:30:45.989677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.989568    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerDied","Data":"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"}
Nov 01 20:30:45.989864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:45.989691    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d" event={"ID":"c9218bd9-0c0f-48ab-88be-c87de234f890","Type":"ContainerDied","Data":"923cc60bd69b8d70d643eae641b68a436bc6e1342d6a42011b6e52bf3d64d08d"}
Nov 01 20:30:46.005155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.005052    5041 scope.go:122] "RemoveContainer" containerID="8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"
Nov 01 20:30:46.005440 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:46.005408    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f\": container with ID starting with 8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f not found: ID does not exist" containerID="8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"
Nov 01 20:30:46.005698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.005580    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f"} err="failed to get container status \"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f\": rpc error: code = NotFound desc = could not find container \"8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f\": container with ID starting with 8012c0650899917216478f6f2d7afe345397664a19fa363ca3c4897641e8bb2f not found: ID does not exist"
Nov 01 20:30:46.005698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.005614    5041 scope.go:122] "RemoveContainer" containerID="b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"
Nov 01 20:30:46.017834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.017810    5041 scope.go:122] "RemoveContainer" containerID="b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"
Nov 01 20:30:46.030688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.030667    5041 scope.go:122] "RemoveContainer" containerID="b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"
Nov 01 20:30:46.031048 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:46.031022    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4\": container with ID starting with b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4 not found: ID does not exist" containerID="b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"
Nov 01 20:30:46.031138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.031060    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4"} err="failed to get container status \"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4\": rpc error: code = NotFound desc = could not find container \"b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4\": container with ID starting with b7f46edef0ab0df96301d63b28ee460a194e10824bb67d8910d30417e77e85f4 not found: ID does not exist"
Nov 01 20:30:46.031138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.031091    5041 scope.go:122] "RemoveContainer" containerID="b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"
Nov 01 20:30:46.031416 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:46.031387    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e\": container with ID starting with b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e not found: ID does not exist" containerID="b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"
Nov 01 20:30:46.031486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.031425    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e"} err="failed to get container status \"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e\": rpc error: code = NotFound desc = could not find container \"b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e\": container with ID starting with b51096c6d6d0a62bb52dc14153e9e2f2bbbd5135f8c67ea4f507d72d06b06b6e not found: ID does not exist"
Nov 01 20:30:46.037393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037372    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.037511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037425    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.037511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037481    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.037511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037501    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.037680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037566    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.037680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037598    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.037811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037672    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.037877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.037944    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038013    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038048    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038238    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038588    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038590    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038662    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038738    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038850    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038893    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.038908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039092    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039220    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039441    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039425    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\") pod \"a4d0f35a-b93d-4245-8745-7f93722536cd\" (UID: \"a4d0f35a-b93d-4245-8745-7f93722536cd\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039520    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039590    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\") pod \"c9218bd9-0c0f-48ab-88be-c87de234f890\" (UID: \"c9218bd9-0c0f-48ab-88be-c87de234f890\") "
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039727    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.039972    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040374    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040400    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d0f35a-b93d-4245-8745-7f93722536cd-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040416    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040430    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040445    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040459    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040488    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040503    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.040518    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.041727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.041680    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:46.043071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.041916    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:46.043071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.042096    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:46.043071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.042168    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "kube-api-access-svn58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:46.043071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.042219    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.043491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.043463    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo" pod "a4d0f35a-b93d-4245-8745-7f93722536cd" (UID: "a4d0f35a-b93d-4245-8745-7f93722536cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:30:46.043727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.043704    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:30:46.043853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.043742    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "kube-api-access-48thb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:46.044106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.044089    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy" pod "c9218bd9-0c0f-48ab-88be-c87de234f890" (UID: "c9218bd9-0c0f-48ab-88be-c87de234f890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.141999    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142032    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48thb\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-kube-api-access-48thb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142042    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svn58\" (UniqueName: \"kubernetes.io/projected/a4d0f35a-b93d-4245-8745-7f93722536cd-kube-api-access-svn58\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142051    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142060    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142068    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142078    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9218bd9-0c0f-48ab-88be-c87de234f890-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142087    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d0f35a-b93d-4245-8745-7f93722536cd-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.142481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.142095    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9218bd9-0c0f-48ab-88be-c87de234f890-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:46.315672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.315640    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:30:46.319537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.319513    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bccce1-openshift-default-79b58b586f-99z9d"]
Nov 01 20:30:46.332403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.332373    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:30:46.336085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.336059    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d2dc0ad5-openshift-default-86bdfb9469-mgfvz"]
Nov 01 20:30:46.429453 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.429370    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d0f35a-b93d-4245-8745-7f93722536cd" path="/var/lib/kubelet/pods/a4d0f35a-b93d-4245-8745-7f93722536cd/volumes"
Nov 01 20:30:46.429932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:46.429916    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" path="/var/lib/kubelet/pods/c9218bd9-0c0f-48ab-88be-c87de234f890/volumes"
Nov 01 20:30:50.141587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.141553    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"]
Nov 01 20:30:50.142370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142339    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.142370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142366    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142395    5041 cpu_manager.go:414] "containerMap: removing container" podUID="12392895-6c60-46a2-b646-09f9efe18b4a" containerName="revoke-keys"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142403    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12392895-6c60-46a2-b646-09f9efe18b4a" containerName="revoke-keys"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142429    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a4d0f35a-b93d-4245-8745-7f93722536cd" containerName="istio-proxy"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142438    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d0f35a-b93d-4245-8745-7f93722536cd" containerName="istio-proxy"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142465    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.142597 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142473    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.142838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142640    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.142838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142653    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="12392895-6c60-46a2-b646-09f9efe18b4a" containerName="revoke-keys"
Nov 01 20:30:50.142838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142661    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d0f35a-b93d-4245-8745-7f93722536cd" containerName="istio-proxy"
Nov 01 20:30:50.142838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.142670    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9218bd9-0c0f-48ab-88be-c87de234f890" containerName="istio-proxy"
Nov 01 20:30:50.148114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.148094    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.152342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.152316    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:30:50.152502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.152360    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:30:50.152588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.152557    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:30:50.152651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.152623    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:30:50.154915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.154891    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"]
Nov 01 20:30:50.295325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.295285    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9rq\" (UniqueName: \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.295558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.295515    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.295626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.295581    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.396815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.396699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.396815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.396763    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.397007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.396883    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9rq\" (UniqueName: \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.397395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.397363    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.399225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.399200    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.404510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.404485    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9rq\" (UniqueName: \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\") pod \"maas-api-revoke-sub-daa8ab57-tj882\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.467694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.467664    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:50.749321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:50.749290    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"]
Nov 01 20:30:50.750228 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:30:50.750200    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3ef7ef_f64a_4f8c_a888_1f21c3e35d27.slice/crio-45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1 WatchSource:0}: Error finding container 45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1: Status 404 returned error can't find the container with id 45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1
Nov 01 20:30:51.032062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:51.031971    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882" event={"ID":"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27","Type":"ContainerStarted","Data":"45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1"}
Nov 01 20:30:52.041211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:52.041175    5041 generic.go:356] "Generic (PLEG): container finished" podID="8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" containerID="235b84bb38582c57fdb66a7ed64e8380b728655a6459f1dfd1d30c1d4e31deaf" exitCode=0
Nov 01 20:30:52.041678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:52.041269    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882" event={"ID":"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27","Type":"ContainerDied","Data":"235b84bb38582c57fdb66a7ed64e8380b728655a6459f1dfd1d30c1d4e31deaf"}
Nov 01 20:30:52.566298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:52.566260    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:30:52.566507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:52.566482    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy" containerID="cri-o://7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6" gracePeriod=30
Nov 01 20:30:53.316706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.316631    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:53.440404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.440370    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\") pod \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") "
Nov 01 20:30:53.440622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.440454    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\") pod \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") "
Nov 01 20:30:53.440622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.440521    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\" (UniqueName: \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\") pod \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\" (UID: \"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27\") "
Nov 01 20:30:53.440868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.440833    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca" pod "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" (UID: "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:53.441141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.441116    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:53.442614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.442581    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca" pod "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" (UID: "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:30:53.442725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.442644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq" pod "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" (UID: "8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27"). InnerVolumeSpecName "kube-api-access-wh9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:53.542792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.542735    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:53.542969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:53.542798    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh9rq\" (UniqueName: \"kubernetes.io/projected/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27-kube-api-access-wh9rq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:54.057905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:54.057877    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"
Nov 01 20:30:54.058093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:54.057861    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882" event={"ID":"8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27","Type":"ContainerDied","Data":"45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1"}
Nov 01 20:30:54.058093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:54.058001    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b27de7301588c8b8cdc4cc94ab344029df393d2f5826f242fc0251b25b44d1"
Nov 01 20:30:57.857484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.857455    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:30:57.884566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.884534    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:30:57.889844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.884756    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy" containerID="cri-o://93f60a129f803eefdeb24d642577a65b15147f3c79603294da991e4ad83c50bc" gracePeriod=30
Nov 01 20:30:57.895502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895467    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.895657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895529    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.895657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895556    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.895657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895629    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.896038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.896038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895745    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.896038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895765    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.896038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895865    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.896038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.895935    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\") pod \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\" (UID: \"4f1a442e-4b5e-43af-be3f-629d9b16aa70\") "
Nov 01 20:30:57.897225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.896678    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:57.897225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.896827    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:57.897225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.896961    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:57.897225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.896995    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:30:57.897225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.897181    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:57.905973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.905938    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:30:57.906129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.906025    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:30:57.907932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.907908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "kube-api-access-vg6p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:57.916524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.916490    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token" pod "4f1a442e-4b5e-43af-be3f-629d9b16aa70" (UID: "4f1a442e-4b5e-43af-be3f-629d9b16aa70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:30:57.997910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997868    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.997910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997909    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997925    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6p6\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-kube-api-access-vg6p6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997941    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997955    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997970    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997983    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.997996    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f1a442e-4b5e-43af-be3f-629d9b16aa70-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:57.998104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:57.998010    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f1a442e-4b5e-43af-be3f-629d9b16aa70-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:30:58.091792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.091677    5041 generic.go:356] "Generic (PLEG): container finished" podID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerID="7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6" exitCode=0
Nov 01 20:30:58.091792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.091727    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"
Nov 01 20:30:58.091792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.091761    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerDied","Data":"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"}
Nov 01 20:30:58.092083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.091838    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k" event={"ID":"4f1a442e-4b5e-43af-be3f-629d9b16aa70","Type":"ContainerDied","Data":"217412f2d92a5de9c5bd0de92c38ce1f3ca8f87b44b2be2da2a2ce82d1733003"}
Nov 01 20:30:58.092083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.091866    5041 scope.go:122] "RemoveContainer" containerID="7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"
Nov 01 20:30:58.103959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.103938    5041 scope.go:122] "RemoveContainer" containerID="7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"
Nov 01 20:30:58.116081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.116060    5041 scope.go:122] "RemoveContainer" containerID="7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"
Nov 01 20:30:58.116387 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:58.116365    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6\": container with ID starting with 7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6 not found: ID does not exist" containerID="7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"
Nov 01 20:30:58.116481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.116399    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6"} err="failed to get container status \"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6\": rpc error: code = NotFound desc = could not find container \"7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6\": container with ID starting with 7792aa16ce75b03b5a6fabe5a94fa05ae0b20daba7c38a3859d62ed95a8653c6 not found: ID does not exist"
Nov 01 20:30:58.116481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.116424    5041 scope.go:122] "RemoveContainer" containerID="7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"
Nov 01 20:30:58.116687 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:58.116661    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3\": container with ID starting with 7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3 not found: ID does not exist" containerID="7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"
Nov 01 20:30:58.116909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.116693    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3"} err="failed to get container status \"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3\": rpc error: code = NotFound desc = could not find container \"7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3\": container with ID starting with 7eef0c49bdc99c7c6af9f62ffef1c9c753cdc59801d397284a6c45a530867bd3 not found: ID does not exist"
Nov 01 20:30:58.118311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.118292    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:30:58.122257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.122231    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-9be196-openshift-default-df667f8b9-zbn7k"]
Nov 01 20:30:58.428841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:30:58.428749    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" path="/var/lib/kubelet/pods/4f1a442e-4b5e-43af-be3f-629d9b16aa70/volumes"
Nov 01 20:30:58.541715 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:30:58.541679    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.626205    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"]
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627130    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" containerName="revoke-keys"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627152    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" containerName="revoke-keys"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627194    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627202    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627230    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627238    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627394    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" containerName="revoke-keys"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627412    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.629444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.627423    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1a442e-4b5e-43af-be3f-629d9b16aa70" containerName="istio-proxy"
Nov 01 20:31:00.638866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.636521    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.648989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.648959    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:31:00.654800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.650971    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:31:00.665907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.665458    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:31:00.666101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.665557    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:31:00.712341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.712297    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"]
Nov 01 20:31:00.739758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.739661    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.740124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.740102    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfp5\" (UniqueName: \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.740386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.740366    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.842017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.841971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.842202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.842115    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfp5\" (UniqueName: \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.842202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.842192    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.842853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.842822    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.844727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.844698    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:00.850643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:00.850617    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfp5\" (UniqueName: \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\") pod \"maas-api-revoke-sub-6de262a6-7jqz5\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:01.074828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:01.074758    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:01.364014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:01.363922    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"]
Nov 01 20:31:01.372110 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:31:01.372067    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8888f1ef_dde2_4066_8889_29b4f469d157.slice/crio-94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199 WatchSource:0}: Error finding container 94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199: Status 404 returned error can't find the container with id 94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199
Nov 01 20:31:02.125862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:02.125815    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5" event={"ID":"8888f1ef-dde2-4066-8889-29b4f469d157","Type":"ContainerStarted","Data":"f4168f7cc2beb295556a27df8a2c5647c74d428ecaac02530d80bc9f49af6de7"}
Nov 01 20:31:02.126281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:02.125882    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5" event={"ID":"8888f1ef-dde2-4066-8889-29b4f469d157","Type":"ContainerStarted","Data":"94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199"}
Nov 01 20:31:02.160930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:02.160878    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5" podStartSLOduration=2.160864264 podStartE2EDuration="2.160864264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:02.158094749 +0000 UTC m=+1949.879173426" watchObservedRunningTime="2026-11-01 20:31:02.160864264 +0000 UTC m=+1949.881942940"
Nov 01 20:31:03.135462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.135431    5041 generic.go:356] "Generic (PLEG): container finished" podID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerID="93f60a129f803eefdeb24d642577a65b15147f3c79603294da991e4ad83c50bc" exitCode=0
Nov 01 20:31:03.135924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.135510    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" event={"ID":"b94f02de-c02d-419b-a4e4-422a55ad2bba","Type":"ContainerDied","Data":"93f60a129f803eefdeb24d642577a65b15147f3c79603294da991e4ad83c50bc"}
Nov 01 20:31:03.137297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.137277    5041 generic.go:356] "Generic (PLEG): container finished" podID="8888f1ef-dde2-4066-8889-29b4f469d157" containerID="f4168f7cc2beb295556a27df8a2c5647c74d428ecaac02530d80bc9f49af6de7" exitCode=0
Nov 01 20:31:03.137428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.137358    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5" event={"ID":"8888f1ef-dde2-4066-8889-29b4f469d157","Type":"ContainerDied","Data":"f4168f7cc2beb295556a27df8a2c5647c74d428ecaac02530d80bc9f49af6de7"}
Nov 01 20:31:03.212008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.211972    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:31:03.285933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.285892    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.285976    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286041    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286073    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286112    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286211    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286263    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:03.286304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286286    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:03.286584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286356    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286404    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\") pod \"b94f02de-c02d-419b-a4e4-422a55ad2bba\" (UID: \"b94f02de-c02d-419b-a4e4-422a55ad2bba\") "
Nov 01 20:31:03.286584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286526    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:03.286876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.286846    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:03.287034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287007    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:03.287308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287273    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.287464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287445    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.287600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287584    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f02de-c02d-419b-a4e4-422a55ad2bba-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.287712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287699    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.287834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.287818    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.289127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.289099    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "kube-api-access-swrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:03.289127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.289119    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:31:03.289427 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.289404    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:03.289510 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.289453    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token" pod "b94f02de-c02d-419b-a4e4-422a55ad2bba" (UID: "b94f02de-c02d-419b-a4e4-422a55ad2bba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:03.388933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.388894    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swrdb\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-kube-api-access-swrdb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.388933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.388929    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.388933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.388939    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:03.389151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:03.388947    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f02de-c02d-419b-a4e4-422a55ad2bba-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:04.146940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.146908    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"
Nov 01 20:31:04.146940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.146924    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b" event={"ID":"b94f02de-c02d-419b-a4e4-422a55ad2bba","Type":"ContainerDied","Data":"b23ea4c0554691afdf08836def8cd8fac4606ae17179e01e957704e2c269f59a"}
Nov 01 20:31:04.147485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.146989    5041 scope.go:122] "RemoveContainer" containerID="93f60a129f803eefdeb24d642577a65b15147f3c79603294da991e4ad83c50bc"
Nov 01 20:31:04.180871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.180449    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:31:04.186602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.186563    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-hwh5b"]
Nov 01 20:31:04.327395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.327369    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:04.401024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.400930    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\") pod \"8888f1ef-dde2-4066-8889-29b4f469d157\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") "
Nov 01 20:31:04.401024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.400987    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\" (UniqueName: \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\") pod \"8888f1ef-dde2-4066-8889-29b4f469d157\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") "
Nov 01 20:31:04.401262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.401028    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\") pod \"8888f1ef-dde2-4066-8889-29b4f469d157\" (UID: \"8888f1ef-dde2-4066-8889-29b4f469d157\") "
Nov 01 20:31:04.401395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.401371    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca" pod "8888f1ef-dde2-4066-8889-29b4f469d157" (UID: "8888f1ef-dde2-4066-8889-29b4f469d157"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:04.401557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.401540    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:04.403144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.403120    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5" pod "8888f1ef-dde2-4066-8889-29b4f469d157" (UID: "8888f1ef-dde2-4066-8889-29b4f469d157"). InnerVolumeSpecName "kube-api-access-7gfp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:04.403258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.403236    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca" pod "8888f1ef-dde2-4066-8889-29b4f469d157" (UID: "8888f1ef-dde2-4066-8889-29b4f469d157"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:04.430882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.430852    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" path="/var/lib/kubelet/pods/b94f02de-c02d-419b-a4e4-422a55ad2bba/volumes"
Nov 01 20:31:04.503252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.503207    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gfp5\" (UniqueName: \"kubernetes.io/projected/8888f1ef-dde2-4066-8889-29b4f469d157-kube-api-access-7gfp5\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:04.503252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:04.503247    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8888f1ef-dde2-4066-8889-29b4f469d157-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:05.157695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:05.157667    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"
Nov 01 20:31:05.158130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:05.157696    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5" event={"ID":"8888f1ef-dde2-4066-8889-29b4f469d157","Type":"ContainerDied","Data":"94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199"}
Nov 01 20:31:05.158130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:05.157739    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94797d9c96146823ebd243fc9f1a23f6d1cea345f0ce4424d7d6fd554c543199"
Nov 01 20:31:12.767335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:12.767296    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:31:13.021934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:13.021846    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"]
Nov 01 20:31:13.025566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:13.025538    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1ff52b4-w4dzq"]
Nov 01 20:31:14.430834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:14.430773    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90" path="/var/lib/kubelet/pods/bc6972e9-34c1-4cf8-8c10-ad5f3c2cde90/volumes"
Nov 01 20:31:23.032394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:23.032344    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"]
Nov 01 20:31:23.039672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:23.039630    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7e8c515-52vbp"]
Nov 01 20:31:24.429975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:24.429884    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211fed86-d69c-49d4-8c03-56b1cf7a8bcc" path="/var/lib/kubelet/pods/211fed86-d69c-49d4-8c03-56b1cf7a8bcc/volumes"
Nov 01 20:31:27.799174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:27.799133    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main" containerID="cri-o://2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb" gracePeriod=45
Nov 01 20:31:28.114697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.114670    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:31:28.256580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256549    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.256758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256691    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.256758 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256735    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\" (UniqueName: \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.256874 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256772    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.256937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256889    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.256937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.256928    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\") pod \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\" (UID: \"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad\") "
Nov 01 20:31:28.257127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257069    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:28.257127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257084    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:28.257400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257377    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:28.257491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257473    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.257554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257496    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.257554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.257506    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.259182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.259160    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "kube-api-access-jqmlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:28.259368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.259344    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:28.259368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.259353    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs" pod "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" (UID: "782e84f8-dd58-4e2e-af9d-fdf5038bb8ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:28.358677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.358587    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmlq\" (UniqueName: \"kubernetes.io/projected/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-kube-api-access-jqmlq\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.358677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.358614    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.358677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.358625    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:28.359466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.359446    5041 generic.go:356] "Generic (PLEG): container finished" podID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerID="2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb" exitCode=0
Nov 01 20:31:28.359525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.359493    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" event={"ID":"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad","Type":"ContainerDied","Data":"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"}
Nov 01 20:31:28.359525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.359502    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"
Nov 01 20:31:28.359525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.359520    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" event={"ID":"782e84f8-dd58-4e2e-af9d-fdf5038bb8ad","Type":"ContainerDied","Data":"b4497dd41dc11acf660c7ed31380b3ee0d660c22c3d380dd36cf4b060e414479"}
Nov 01 20:31:28.359622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.359538    5041 scope.go:122] "RemoveContainer" containerID="2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"
Nov 01 20:31:28.370896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.370874    5041 scope.go:122] "RemoveContainer" containerID="2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"
Nov 01 20:31:28.371188 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:31:28.371165    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb\": container with ID starting with 2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb not found: ID does not exist" containerID="2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"
Nov 01 20:31:28.371248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.371197    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb"} err="failed to get container status \"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb\": rpc error: code = NotFound desc = could not find container \"2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb\": container with ID starting with 2719049353f384eb02e15664f7e8f4ab3b251af2a93934152ebe10e0865aa6cb not found: ID does not exist"
Nov 01 20:31:28.383902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.383875    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:31:28.387571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.387545    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk"]
Nov 01 20:31:28.428667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:28.428638    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" path="/var/lib/kubelet/pods/782e84f8-dd58-4e2e-af9d-fdf5038bb8ad/volumes"
Nov 01 20:31:28.555185 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:31:28.555148    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:31:29.103289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.099639    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-036c69/sub-test-model-036c69-kserve-66cbb89ff-j6fxk" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:31:29.748826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.748791    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"]
Nov 01 20:31:29.749508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749489    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8888f1ef-dde2-4066-8889-29b4f469d157" containerName="revoke-keys"
Nov 01 20:31:29.749508 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749508    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8888f1ef-dde2-4066-8889-29b4f469d157" containerName="revoke-keys"
Nov 01 20:31:29.749671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749539    5041 cpu_manager.go:414] "containerMap: removing container" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main"
Nov 01 20:31:29.749671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749545    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main"
Nov 01 20:31:29.749671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749559    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy"
Nov 01 20:31:29.749671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749564    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy"
Nov 01 20:31:29.749913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749676    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="782e84f8-dd58-4e2e-af9d-fdf5038bb8ad" containerName="main"
Nov 01 20:31:29.749913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749691    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94f02de-c02d-419b-a4e4-422a55ad2bba" containerName="istio-proxy"
Nov 01 20:31:29.749913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.749709    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8888f1ef-dde2-4066-8889-29b4f469d157" containerName="revoke-keys"
Nov 01 20:31:29.754917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.754893    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.758760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.758732    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:31:29.760312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.760289    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:31:29.760414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.760396    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:31:29.760463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.760434    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:31:29.760641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.760617    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"]
Nov 01 20:31:29.877313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.877274    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.877534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.877495    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhzt\" (UniqueName: \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.877534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.877525    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.978709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.978670    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.978935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.978812    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.979013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.978956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhzt\" (UniqueName: \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.979361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.979335    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.981240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.981218    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:29.986851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:29.986829    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhzt\" (UniqueName: \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\") pod \"maas-api-revoke-sub-91086483-jwtgb\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:30.074974 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:30.074940    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:30.357200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:30.357175    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"]
Nov 01 20:31:30.358030 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:31:30.358003    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f63d6f_26f0_4586_a9f7_64d722b5f021.slice/crio-c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9 WatchSource:0}: Error finding container c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9: Status 404 returned error can't find the container with id c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9
Nov 01 20:31:30.376966 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:30.376938    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb" event={"ID":"63f63d6f-26f0-4586-a9f7-64d722b5f021","Type":"ContainerStarted","Data":"c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9"}
Nov 01 20:31:31.386865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:31.386827    5041 generic.go:356] "Generic (PLEG): container finished" podID="63f63d6f-26f0-4586-a9f7-64d722b5f021" containerID="874b1c714c9da6b9aecad9f87a88a1b54c79cc1c91676bc15747ddf725148363" exitCode=0
Nov 01 20:31:31.386865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:31.386870    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb" event={"ID":"63f63d6f-26f0-4586-a9f7-64d722b5f021","Type":"ContainerDied","Data":"874b1c714c9da6b9aecad9f87a88a1b54c79cc1c91676bc15747ddf725148363"}
Nov 01 20:31:32.574423 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.574397    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:32.714652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.714565    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\" (UniqueName: \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\") pod \"63f63d6f-26f0-4586-a9f7-64d722b5f021\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") "
Nov 01 20:31:32.714652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.714604    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\") pod \"63f63d6f-26f0-4586-a9f7-64d722b5f021\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") "
Nov 01 20:31:32.714652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.714629    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\") pod \"63f63d6f-26f0-4586-a9f7-64d722b5f021\" (UID: \"63f63d6f-26f0-4586-a9f7-64d722b5f021\") "
Nov 01 20:31:32.715119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.715089    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca" pod "63f63d6f-26f0-4586-a9f7-64d722b5f021" (UID: "63f63d6f-26f0-4586-a9f7-64d722b5f021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:32.717019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.716995    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt" pod "63f63d6f-26f0-4586-a9f7-64d722b5f021" (UID: "63f63d6f-26f0-4586-a9f7-64d722b5f021"). InnerVolumeSpecName "kube-api-access-rqhzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:32.717188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.717165    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca" pod "63f63d6f-26f0-4586-a9f7-64d722b5f021" (UID: "63f63d6f-26f0-4586-a9f7-64d722b5f021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:32.825129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.825086    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqhzt\" (UniqueName: \"kubernetes.io/projected/63f63d6f-26f0-4586-a9f7-64d722b5f021-kube-api-access-rqhzt\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:32.825396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.825371    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:32.825396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:32.831744    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f63d6f-26f0-4586-a9f7-64d722b5f021-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:33.402246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:33.402212    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"
Nov 01 20:31:33.402441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:33.402246    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb" event={"ID":"63f63d6f-26f0-4586-a9f7-64d722b5f021","Type":"ContainerDied","Data":"c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9"}
Nov 01 20:31:33.402441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:33.402315    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c460fa34dae9ac2a1c460aec172199b12f1f2670f95cfe4a9cc077c59a4035b9"
Nov 01 20:31:34.355787 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:34.355737    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:31:36.337019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:36.336989    5041 scope.go:122] "RemoveContainer" containerID="91baeb0089b48191bcfa815ad83740a317d4a29b9a128ca78a963addaf009c63"
Nov 01 20:31:36.347433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:36.347414    5041 scope.go:122] "RemoveContainer" containerID="29c096740696ba49871108a822becb9b9ce8f8a83d75ef0b538c1b5afbdeff6c"
Nov 01 20:31:36.356698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:36.356679    5041 scope.go:122] "RemoveContainer" containerID="d90bb41ea2f53cfcb8c92a0c33807986da8495e23044eac65c31697fed7ae213"
Nov 01 20:31:36.365747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:36.365725    5041 scope.go:122] "RemoveContainer" containerID="aaba946aa5dc99ed52f88f540862a0c9b962d18a958b41266c5cd093a979a3d3"
Nov 01 20:31:39.982278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:39.982237    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:31:39.999836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:39.999801    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:31:41.247794 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.247754    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"]
Nov 01 20:31:41.248340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.248326    5041 cpu_manager.go:414] "containerMap: removing container" podUID="63f63d6f-26f0-4586-a9f7-64d722b5f021" containerName="revoke-keys"
Nov 01 20:31:41.248390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.248340    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f63d6f-26f0-4586-a9f7-64d722b5f021" containerName="revoke-keys"
Nov 01 20:31:41.248456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.248447    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f63d6f-26f0-4586-a9f7-64d722b5f021" containerName="revoke-keys"
Nov 01 20:31:41.254178 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.254156    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.258288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.258265    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:31:41.258452 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.258319    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:31:41.261664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.261628    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"]
Nov 01 20:31:41.266873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.266851    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:31:41.267019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.266984    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:31:41.339503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.339471    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhdj\" (UniqueName: \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.339689 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.339609    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.339753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.339737    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.441202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.441166    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhdj\" (UniqueName: \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.441385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.441262    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.441385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.441351    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.442130 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.442099    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.443825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.443798    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.449444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.449419    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhdj\" (UniqueName: \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\") pod \"maas-api-revoke-sub-621b3900-mtxbt\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.570823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.570788    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:41.859198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:41.859162    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"]
Nov 01 20:31:41.859365 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:31:41.859341    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04575f7d_d2ed_4642_90a4_fb38871a3d16.slice/crio-8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190 WatchSource:0}: Error finding container 8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190: Status 404 returned error can't find the container with id 8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190
Nov 01 20:31:42.475586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:42.475491    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt" event={"ID":"04575f7d-d2ed-4642-90a4-fb38871a3d16","Type":"ContainerStarted","Data":"52d8e80bafcd3e7162a47cce86dff0065fbf1bdd7c0d93ea3fcede14be41b344"}
Nov 01 20:31:42.475586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:42.475552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt" event={"ID":"04575f7d-d2ed-4642-90a4-fb38871a3d16","Type":"ContainerStarted","Data":"8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190"}
Nov 01 20:31:42.496556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:42.496507    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt" podStartSLOduration=1.496490809 podStartE2EDuration="1.496490809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:31:42.491278873 +0000 UTC m=+1990.212357545" watchObservedRunningTime="2026-11-01 20:31:42.496490809 +0000 UTC m=+1990.217569486"
Nov 01 20:31:43.484316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:43.484279    5041 generic.go:356] "Generic (PLEG): container finished" podID="04575f7d-d2ed-4642-90a4-fb38871a3d16" containerID="52d8e80bafcd3e7162a47cce86dff0065fbf1bdd7c0d93ea3fcede14be41b344" exitCode=0
Nov 01 20:31:43.484760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:43.484326    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt" event={"ID":"04575f7d-d2ed-4642-90a4-fb38871a3d16","Type":"ContainerDied","Data":"52d8e80bafcd3e7162a47cce86dff0065fbf1bdd7c0d93ea3fcede14be41b344"}
Nov 01 20:31:44.021722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.021682    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"]
Nov 01 20:31:44.025550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.025522    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bc02c6-6g6xj"]
Nov 01 20:31:44.429198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.429163    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada3a6cb-0068-4557-87f2-77d1167fbf6f" path="/var/lib/kubelet/pods/ada3a6cb-0068-4557-87f2-77d1167fbf6f/volumes"
Nov 01 20:31:44.645968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.645943    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:44.683695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.683620    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\" (UniqueName: \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\") pod \"04575f7d-d2ed-4642-90a4-fb38871a3d16\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") "
Nov 01 20:31:44.683888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.683724    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\") pod \"04575f7d-d2ed-4642-90a4-fb38871a3d16\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") "
Nov 01 20:31:44.683888 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.683878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\") pod \"04575f7d-d2ed-4642-90a4-fb38871a3d16\" (UID: \"04575f7d-d2ed-4642-90a4-fb38871a3d16\") "
Nov 01 20:31:44.684185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.684153    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca" pod "04575f7d-d2ed-4642-90a4-fb38871a3d16" (UID: "04575f7d-d2ed-4642-90a4-fb38871a3d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:31:44.684469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.684450    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:44.686823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.686538    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca" pod "04575f7d-d2ed-4642-90a4-fb38871a3d16" (UID: "04575f7d-d2ed-4642-90a4-fb38871a3d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:44.686823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.686545    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj" pod "04575f7d-d2ed-4642-90a4-fb38871a3d16" (UID: "04575f7d-d2ed-4642-90a4-fb38871a3d16"). InnerVolumeSpecName "kube-api-access-wdhdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:44.785464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.785430    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04575f7d-d2ed-4642-90a4-fb38871a3d16-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:44.785464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.785459    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhdj\" (UniqueName: \"kubernetes.io/projected/04575f7d-d2ed-4642-90a4-fb38871a3d16-kube-api-access-wdhdj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:44.983097 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:44.982984    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing" containerID="cri-o://61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b" gracePeriod=25
Nov 01 20:31:45.000724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.000666    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing" containerID="cri-o://e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6" gracePeriod=25
Nov 01 20:31:45.499946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.499913    5041 generic.go:356] "Generic (PLEG): container finished" podID="9d8742c6-60a8-4b53-b58e-ced648946d28" containerID="4f913d478545e592d97ce9810f14ee1922386d90de87aa21c0b2d34bab4524ff" exitCode=1
Nov 01 20:31:45.500127 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.499990    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k" event={"ID":"9d8742c6-60a8-4b53-b58e-ced648946d28","Type":"ContainerDied","Data":"4f913d478545e592d97ce9810f14ee1922386d90de87aa21c0b2d34bab4524ff"}
Nov 01 20:31:45.500409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.500392    5041 scope.go:122] "RemoveContainer" containerID="4f913d478545e592d97ce9810f14ee1922386d90de87aa21c0b2d34bab4524ff"
Nov 01 20:31:45.501515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.501493    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"
Nov 01 20:31:45.501632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.501560    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt" event={"ID":"04575f7d-d2ed-4642-90a4-fb38871a3d16","Type":"ContainerDied","Data":"8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190"}
Nov 01 20:31:45.501632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:45.501596    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a462fe1a78f18da0bbd761f996bb9869ad13e6c8f4b2bcef27fb53f9e2b5190"
Nov 01 20:31:46.512634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:46.512595    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k" event={"ID":"9d8742c6-60a8-4b53-b58e-ced648946d28","Type":"ContainerStarted","Data":"3233a37ad61abc84e7c6aac91b26454f965a8d368d41b34e174c0c44277be6ef"}
Nov 01 20:31:46.513214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:46.512966    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:31:49.377736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.377701    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" containerID="cri-o://537078708ca0a6af3f0252e0c67c4608c462caa54d6bd5c797b4b3859bfde31a" gracePeriod=45
Nov 01 20:31:49.425488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.425448    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Nov 01 20:31:49.549754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.549720    5041 generic.go:356] "Generic (PLEG): container finished" podID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerID="537078708ca0a6af3f0252e0c67c4608c462caa54d6bd5c797b4b3859bfde31a" exitCode=0
Nov 01 20:31:49.549933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.549823    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" event={"ID":"c63cfe70-ac77-474b-a224-5d2450af17b4","Type":"ContainerDied","Data":"537078708ca0a6af3f0252e0c67c4608c462caa54d6bd5c797b4b3859bfde31a"}
Nov 01 20:31:49.672363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.672334    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768335    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768423    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768455    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768556    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\" (UniqueName: \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768631    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.768800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768695    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.769243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768851    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\") pod \"c63cfe70-ac77-474b-a224-5d2450af17b4\" (UID: \"c63cfe70-ac77-474b-a224-5d2450af17b4\") "
Nov 01 20:31:49.769243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.768937    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.769243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.769121    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.769397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.769336    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.769397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.769357    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.769397 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.769372    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.777858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.777442    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "kube-api-access-5cm7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:31:49.777858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.777443    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:31:49.779940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.779912    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs" pod "c63cfe70-ac77-474b-a224-5d2450af17b4" (UID: "c63cfe70-ac77-474b-a224-5d2450af17b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:31:49.871624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.871590    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63cfe70-ac77-474b-a224-5d2450af17b4-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.871624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.871621    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63cfe70-ac77-474b-a224-5d2450af17b4-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:49.871917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:49.871633    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cm7s\" (UniqueName: \"kubernetes.io/projected/c63cfe70-ac77-474b-a224-5d2450af17b4-kube-api-access-5cm7s\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:31:50.558406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:50.558372    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"
Nov 01 20:31:50.558928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:50.558400    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt" event={"ID":"c63cfe70-ac77-474b-a224-5d2450af17b4","Type":"ContainerDied","Data":"50768f96cbdaf8c934dc182ce5780497a171306e154bb23937d3df5ae2be1b71"}
Nov 01 20:31:50.558928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:50.558461    5041 scope.go:122] "RemoveContainer" containerID="537078708ca0a6af3f0252e0c67c4608c462caa54d6bd5c797b4b3859bfde31a"
Nov 01 20:31:50.577500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:50.577429    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:31:50.582370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:50.582342    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/premium-w0-3a6480-kserve-67c95c7964-z9smt"]
Nov 01 20:31:52.021949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:52.021914    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"]
Nov 01 20:31:52.025529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:52.025505    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7888f16-p2tvw"]
Nov 01 20:31:52.428889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:52.428859    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1472eaab-f207-4de5-b138-86210a4ce907" path="/var/lib/kubelet/pods/1472eaab-f207-4de5-b138-86210a4ce907/volumes"
Nov 01 20:31:52.429357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:52.429342    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" path="/var/lib/kubelet/pods/c63cfe70-ac77-474b-a224-5d2450af17b4/volumes"
Nov 01 20:31:53.157634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:53.157594    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:31:53.157634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:53.157620    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:31:57.524532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:57.524498    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-msx7k"
Nov 01 20:31:58.024252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:58.024215    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"]
Nov 01 20:31:58.028096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:58.028073    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee514429-c6thj"]
Nov 01 20:31:58.428972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:31:58.428936    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cae964-2241-42c0-a74f-dc4b9ff3e4ca" path="/var/lib/kubelet/pods/c5cae964-2241-42c0-a74f-dc4b9ff3e4ca/volumes"
Nov 01 20:31:58.545146 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:31:58.545102    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:32:03.158230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:03.158195    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:03.158641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:03.158195    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:32:05.268359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.268320    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:32:05.305073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.305043    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:32:05.374401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.374365    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\" (UniqueName: \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\") pod \"65370231-0b33-48e1-8c1c-e8b23c347555\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") "
Nov 01 20:32:05.374586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.374487    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\" (UniqueName: \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\") pod \"98e334ce-7226-460d-a52f-ac96ebe93f6c\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") "
Nov 01 20:32:05.374682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.374584    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\") pod \"98e334ce-7226-460d-a52f-ac96ebe93f6c\" (UID: \"98e334ce-7226-460d-a52f-ac96ebe93f6c\") "
Nov 01 20:32:05.374682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.374635    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\") pod \"65370231-0b33-48e1-8c1c-e8b23c347555\" (UID: \"65370231-0b33-48e1-8c1c-e8b23c347555\") "
Nov 01 20:32:05.375045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.375013    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume" pod "98e334ce-7226-460d-a52f-ac96ebe93f6c" (UID: "98e334ce-7226-460d-a52f-ac96ebe93f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:05.375189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.375093    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume" pod "65370231-0b33-48e1-8c1c-e8b23c347555" (UID: "65370231-0b33-48e1-8c1c-e8b23c347555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:05.375263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.375200    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e334ce-7226-460d-a52f-ac96ebe93f6c-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:05.375263 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.375220    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65370231-0b33-48e1-8c1c-e8b23c347555-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:05.376625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.376600    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj" pod "65370231-0b33-48e1-8c1c-e8b23c347555" (UID: "65370231-0b33-48e1-8c1c-e8b23c347555"). InnerVolumeSpecName "kube-api-access-fcxfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:05.376721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.376675    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz" pod "98e334ce-7226-460d-a52f-ac96ebe93f6c" (UID: "98e334ce-7226-460d-a52f-ac96ebe93f6c"). InnerVolumeSpecName "kube-api-access-sbmlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:05.476309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.476265    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxfj\" (UniqueName: \"kubernetes.io/projected/65370231-0b33-48e1-8c1c-e8b23c347555-kube-api-access-fcxfj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:05.476309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.476305    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmlz\" (UniqueName: \"kubernetes.io/projected/98e334ce-7226-460d-a52f-ac96ebe93f6c-kube-api-access-sbmlz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:05.671419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.671380    5041 generic.go:356] "Generic (PLEG): container finished" podID="65370231-0b33-48e1-8c1c-e8b23c347555" containerID="e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6" exitCode=0
Nov 01 20:32:05.671634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.671441    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"
Nov 01 20:32:05.671634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.671468    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" event={"ID":"65370231-0b33-48e1-8c1c-e8b23c347555","Type":"ContainerDied","Data":"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"}
Nov 01 20:32:05.671634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.671528    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf" event={"ID":"65370231-0b33-48e1-8c1c-e8b23c347555","Type":"ContainerDied","Data":"7d3e0751bdd44668bb445d21a151db5fe2c948d5fdfe6a56244bb762f5141d31"}
Nov 01 20:32:05.671634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.671555    5041 scope.go:122] "RemoveContainer" containerID="e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"
Nov 01 20:32:05.673025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.673003    5041 generic.go:356] "Generic (PLEG): container finished" podID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerID="61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b" exitCode=0
Nov 01 20:32:05.673113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.673044    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" event={"ID":"98e334ce-7226-460d-a52f-ac96ebe93f6c","Type":"ContainerDied","Data":"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"}
Nov 01 20:32:05.673113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.673062    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"
Nov 01 20:32:05.673113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.673078    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6" event={"ID":"98e334ce-7226-460d-a52f-ac96ebe93f6c","Type":"ContainerDied","Data":"6e468ac187bfb5d6370cfbbf0e7f68cf1593493b131414aee3c51f7347f13b20"}
Nov 01 20:32:05.683396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.683378    5041 scope.go:122] "RemoveContainer" containerID="e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"
Nov 01 20:32:05.683664 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:05.683644    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6\": container with ID starting with e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6 not found: ID does not exist" containerID="e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"
Nov 01 20:32:05.683737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.683670    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6"} err="failed to get container status \"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6\": rpc error: code = NotFound desc = could not find container \"e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6\": container with ID starting with e107899c56b66ccad8cfd05b3a76cff49f8e02070f65a48cdc5296a3995da2c6 not found: ID does not exist"
Nov 01 20:32:05.683737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.683686    5041 scope.go:122] "RemoveContainer" containerID="61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"
Nov 01 20:32:05.693504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.693484    5041 scope.go:122] "RemoveContainer" containerID="61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"
Nov 01 20:32:05.693774 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:05.693752    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b\": container with ID starting with 61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b not found: ID does not exist" containerID="61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"
Nov 01 20:32:05.693885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.693817    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b"} err="failed to get container status \"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b\": rpc error: code = NotFound desc = could not find container \"61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b\": container with ID starting with 61302d435fcbb509f06419d7c9cbc80526277395bfbfed1c444155ed39b5f59b not found: ID does not exist"
Nov 01 20:32:05.702135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.702108    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:32:05.705842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.705821    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-3a6480-d6b5747dc-rdlxf"]
Nov 01 20:32:05.714267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.714244    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:32:05.717528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:05.717509    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3a6480-84cb444cf-2rlt6"]
Nov 01 20:32:06.427872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:06.427843    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" path="/var/lib/kubelet/pods/65370231-0b33-48e1-8c1c-e8b23c347555/volumes"
Nov 01 20:32:06.428297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:06.428282    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" path="/var/lib/kubelet/pods/98e334ce-7226-460d-a52f-ac96ebe93f6c/volumes"
Nov 01 20:32:08.616898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:08.616860    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:32:08.617351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:08.617101    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" containerID="cri-o://909ef718fa6d0c75b274107451f7b14fa04972779e26cf5df1325eb64f43a699" gracePeriod=30
Nov 01 20:32:12.128681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:12.128642    5041 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:32:12.129105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:12.128731    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:32:13.103473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.103388    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:32:13.103682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.103654    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy" containerID="cri-o://0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088" gracePeriod=30
Nov 01 20:32:13.738994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.738961    5041 generic.go:356] "Generic (PLEG): container finished" podID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerID="909ef718fa6d0c75b274107451f7b14fa04972779e26cf5df1325eb64f43a699" exitCode=0
Nov 01 20:32:13.739450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.739030    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" event={"ID":"dfe9fad5-626b-452a-b794-3c1c0cf872c3","Type":"ContainerDied","Data":"909ef718fa6d0c75b274107451f7b14fa04972779e26cf5df1325eb64f43a699"}
Nov 01 20:32:13.918990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.918964    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:32:13.980095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980119    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980169    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980218    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980260    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980296    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980358    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980419    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980503    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\") pod \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\" (UID: \"dfe9fad5-626b-452a-b794-3c1c0cf872c3\") "
Nov 01 20:32:13.980713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.980618    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.981075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.981053    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:13.981237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.981096    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.981357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.981174    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.981465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.981253    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.981465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.981456    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:13.983513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.982951    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:13.983513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.983452    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "kube-api-access-lbbbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:13.983736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.983711    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:13.983853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:13.983714    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo" pod "dfe9fad5-626b-452a-b794-3c1c0cf872c3" (UID: "dfe9fad5-626b-452a-b794-3c1c0cf872c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:14.082398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082361    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082394    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082404    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082413    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082424    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082434    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbbs\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-kube-api-access-lbbbs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082442    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.082656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.082452    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe9fad5-626b-452a-b794-3c1c0cf872c3-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:14.749346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.749268    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"
Nov 01 20:32:14.749346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.749300    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2" event={"ID":"dfe9fad5-626b-452a-b794-3c1c0cf872c3","Type":"ContainerDied","Data":"101b4c6c0c20aec77d7dcf4616370025d45edf7e5ae1339462e3f50583382a71"}
Nov 01 20:32:14.749915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.749372    5041 scope.go:122] "RemoveContainer" containerID="909ef718fa6d0c75b274107451f7b14fa04972779e26cf5df1325eb64f43a699"
Nov 01 20:32:14.771479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.771442    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:32:14.774856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:14.774831    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-22jj2"]
Nov 01 20:32:16.428556 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:16.428523    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" path="/var/lib/kubelet/pods/dfe9fad5-626b-452a-b794-3c1c0cf872c3/volumes"
Nov 01 20:32:17.742737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.742700    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"]
Nov 01 20:32:17.743345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743328    5041 cpu_manager.go:414] "containerMap: removing container" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy"
Nov 01 20:32:17.743389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743347    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy"
Nov 01 20:32:17.743389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743358    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main"
Nov 01 20:32:17.743389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743362    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main"
Nov 01 20:32:17.743389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743378    5041 cpu_manager.go:414] "containerMap: removing container" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing"
Nov 01 20:32:17.743389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743384    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743417    5041 cpu_manager.go:414] "containerMap: removing container" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743422    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743434    5041 cpu_manager.go:414] "containerMap: removing container" podUID="04575f7d-d2ed-4642-90a4-fb38871a3d16" containerName="revoke-keys"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743439    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04575f7d-d2ed-4642-90a4-fb38871a3d16" containerName="revoke-keys"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743520    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="65370231-0b33-48e1-8c1c-e8b23c347555" containerName="payload-pre-processing"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743529    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="04575f7d-d2ed-4642-90a4-fb38871a3d16" containerName="revoke-keys"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743536    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe9fad5-626b-452a-b794-3c1c0cf872c3" containerName="istio-proxy"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743543    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63cfe70-ac77-474b-a224-5d2450af17b4" containerName="main"
Nov 01 20:32:17.743547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.743550    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e334ce-7226-460d-a52f-ac96ebe93f6c" containerName="payload-processing"
Nov 01 20:32:17.747987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.747970    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.752730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.752701    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:32:17.752952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.752887    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:32:17.758810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.758740    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"]
Nov 01 20:32:17.769062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.769022    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:32:17.769220 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.769082    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:32:17.839461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.839421    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7n6\" (UniqueName: \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.839643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.839491    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.839701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.839678    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.940686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.940638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.940892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.940727    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7n6\" (UniqueName: \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.940892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.940839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.941527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.941496    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.943223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.943200    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:17.948573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:17.948548    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7n6\" (UniqueName: \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\") pod \"maas-api-revoke-sub-edb3d6e8-vltxn\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:18.072268 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.072227    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:18.372374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.372340    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"]
Nov 01 20:32:18.374539 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:32:18.374500    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31b3de5_929e_41fb_a5e0_ce8941dabe10.slice/crio-0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd WatchSource:0}: Error finding container 0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd: Status 404 returned error can't find the container with id 0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd
Nov 01 20:32:18.402196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.402164    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:32:18.550381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550354    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550387    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550422    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550482    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550537    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550567    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550594    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550601    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.550804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550699    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.550804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550700    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.551104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550884    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.551104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.550889    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\") pod \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\" (UID: \"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d\") "
Nov 01 20:32:18.551216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551128    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.551432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551394    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:18.551554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551429    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.551554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551459    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.551554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551473    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.551554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.551486    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.553490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.553465    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:18.553490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.553480    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "kube-api-access-n2p2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:18.553639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.553483    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:18.553639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.553520    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token" pod "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" (UID: "e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:18.652434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.652401    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2p2r\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-kube-api-access-n2p2r\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.652591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.652443    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.652591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.652454    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.652591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.652463    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.652591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.652473    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:18.787606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.787572    5041 generic.go:356] "Generic (PLEG): container finished" podID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerID="0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088" exitCode=0
Nov 01 20:32:18.788131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.787630    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"
Nov 01 20:32:18.788131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.787660    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerDied","Data":"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"}
Nov 01 20:32:18.788131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.787717    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb" event={"ID":"e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d","Type":"ContainerDied","Data":"846e5fd914130d5bd43fe0cf6208c20d0c5bd95835962f03d328fde16dfd8927"}
Nov 01 20:32:18.788131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.787741    5041 scope.go:122] "RemoveContainer" containerID="0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"
Nov 01 20:32:18.789948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.789913    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn" event={"ID":"c31b3de5-929e-41fb-a5e0-ce8941dabe10","Type":"ContainerStarted","Data":"beb53d4a8b7dc9387ee9418a856796961112e8f0cd72a0f729eb498fc572dd0b"}
Nov 01 20:32:18.790081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.789973    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn" event={"ID":"c31b3de5-929e-41fb-a5e0-ce8941dabe10","Type":"ContainerStarted","Data":"0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd"}
Nov 01 20:32:18.802525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.802500    5041 scope.go:122] "RemoveContainer" containerID="ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"
Nov 01 20:32:18.806732 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.806678    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn" podStartSLOduration=1.806659329 podStartE2EDuration="1.806659329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:18.805473629 +0000 UTC m=+2026.526552306" watchObservedRunningTime="2026-11-01 20:32:18.806659329 +0000 UTC m=+2026.527738006"
Nov 01 20:32:18.814205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.814185    5041 scope.go:122] "RemoveContainer" containerID="0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"
Nov 01 20:32:18.814533 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:18.814510    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088\": container with ID starting with 0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088 not found: ID does not exist" containerID="0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"
Nov 01 20:32:18.814644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.814540    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088"} err="failed to get container status \"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088\": rpc error: code = NotFound desc = could not find container \"0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088\": container with ID starting with 0463c67a31569a261468447db60c95c324f87a3d7bc6876114759d5e95d8e088 not found: ID does not exist"
Nov 01 20:32:18.814644 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.814556    5041 scope.go:122] "RemoveContainer" containerID="ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"
Nov 01 20:32:18.814898 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:18.814874    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e\": container with ID starting with ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e not found: ID does not exist" containerID="ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"
Nov 01 20:32:18.814996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.814903    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e"} err="failed to get container status \"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e\": rpc error: code = NotFound desc = could not find container \"ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e\": container with ID starting with ab3e35e41f9d1cc8fa0880ed541366da83b16ba7ab082173c3503784ebce447e not found: ID does not exist"
Nov 01 20:32:18.822030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.822002    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:32:18.825641 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:18.825615    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-jz7fb"]
Nov 01 20:32:19.799711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:19.799669    5041 generic.go:356] "Generic (PLEG): container finished" podID="c31b3de5-929e-41fb-a5e0-ce8941dabe10" containerID="beb53d4a8b7dc9387ee9418a856796961112e8f0cd72a0f729eb498fc572dd0b" exitCode=0
Nov 01 20:32:19.800143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:19.799725    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn" event={"ID":"c31b3de5-929e-41fb-a5e0-ce8941dabe10","Type":"ContainerDied","Data":"beb53d4a8b7dc9387ee9418a856796961112e8f0cd72a0f729eb498fc572dd0b"}
Nov 01 20:32:20.428673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:20.428637    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" path="/var/lib/kubelet/pods/e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d/volumes"
Nov 01 20:32:21.007309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.007282    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:21.120067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.119977    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\") pod \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") "
Nov 01 20:32:21.120246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.120088    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\" (UniqueName: \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\") pod \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") "
Nov 01 20:32:21.120246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.120208    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\") pod \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\" (UID: \"c31b3de5-929e-41fb-a5e0-ce8941dabe10\") "
Nov 01 20:32:21.120396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.120324    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca" pod "c31b3de5-929e-41fb-a5e0-ce8941dabe10" (UID: "c31b3de5-929e-41fb-a5e0-ce8941dabe10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:21.120704 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.120680    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.122351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.122318    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6" pod "c31b3de5-929e-41fb-a5e0-ce8941dabe10" (UID: "c31b3de5-929e-41fb-a5e0-ce8941dabe10"). InnerVolumeSpecName "kube-api-access-pg7n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:21.122461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.122400    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca" pod "c31b3de5-929e-41fb-a5e0-ce8941dabe10" (UID: "c31b3de5-929e-41fb-a5e0-ce8941dabe10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:21.221658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.221618    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7n6\" (UniqueName: \"kubernetes.io/projected/c31b3de5-929e-41fb-a5e0-ce8941dabe10-kube-api-access-pg7n6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.221658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.221653    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b3de5-929e-41fb-a5e0-ce8941dabe10-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:21.816337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.816306    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"
Nov 01 20:32:21.816525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.816332    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn" event={"ID":"c31b3de5-929e-41fb-a5e0-ce8941dabe10","Type":"ContainerDied","Data":"0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd"}
Nov 01 20:32:21.816525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:21.816380    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d788bccdb7cdb281dc897e818770a4a2ab9d3d6fdf103c1f2ce2a013bd083cd"
Nov 01 20:32:22.328215 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:22.328182    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:32:22.328917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:22.328433    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" containerID="cri-o://5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66" gracePeriod=30
Nov 01 20:32:22.348476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:22.348109    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:32:25.902709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:25.902666    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:32:25.903155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:25.902734    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:32:27.593841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.593815    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:32:27.725343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725262    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725307    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725453    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725474    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725527    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725556    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725590    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725634    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725663    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\") pod \"1c276b11-b7a7-4717-b385-9e1c04b1820a\" (UID: \"1c276b11-b7a7-4717-b385-9e1c04b1820a\") "
Nov 01 20:32:27.725985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725818    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.725985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.725883    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.726110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726064    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.726233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726208    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.726233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726219    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:27.726343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726232    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.726343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726268    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.726343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.726287    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.727617 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.727592    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "kube-api-access-85nlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:27.727715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.727663    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:27.728064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.728044    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:27.728230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.728212    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token" pod "1c276b11-b7a7-4717-b385-9e1c04b1820a" (UID: "1c276b11-b7a7-4717-b385-9e1c04b1820a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:27.827496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827454    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.827496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827493    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.827496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827506    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.827934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827519    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c276b11-b7a7-4717-b385-9e1c04b1820a-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.827934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827534    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85nlz\" (UniqueName: \"kubernetes.io/projected/1c276b11-b7a7-4717-b385-9e1c04b1820a-kube-api-access-85nlz\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.827934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.827550    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c276b11-b7a7-4717-b385-9e1c04b1820a-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:27.871331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.871292    5041 generic.go:356] "Generic (PLEG): container finished" podID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerID="5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66" exitCode=0
Nov 01 20:32:27.871522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.871346    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"
Nov 01 20:32:27.871522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.871382    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerDied","Data":"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"}
Nov 01 20:32:27.871522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.871459    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz" event={"ID":"1c276b11-b7a7-4717-b385-9e1c04b1820a","Type":"ContainerDied","Data":"7175b65fb4dd746d93e0410fa4ba4b856ec7db709be46d5c1e89fecbbea85fa3"}
Nov 01 20:32:27.871522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.871498    5041 scope.go:122] "RemoveContainer" containerID="5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"
Nov 01 20:32:27.893509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.893046    5041 scope.go:122] "RemoveContainer" containerID="05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"
Nov 01 20:32:27.912487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.912451    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:32:27.915407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.915382    5041 scope.go:122] "RemoveContainer" containerID="5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"
Nov 01 20:32:27.916112 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:27.916085    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66\": container with ID starting with 5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66 not found: ID does not exist" containerID="5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"
Nov 01 20:32:27.916566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.916127    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66"} err="failed to get container status \"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66\": rpc error: code = NotFound desc = could not find container \"5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66\": container with ID starting with 5d6eecfbee262c21822de566018f8f28088ff15a47f1c63d227f3345aaf3de66 not found: ID does not exist"
Nov 01 20:32:27.916653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.916571    5041 scope.go:122] "RemoveContainer" containerID="05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"
Nov 01 20:32:27.918083 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:27.918057    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a\": container with ID starting with 05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a not found: ID does not exist" containerID="05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"
Nov 01 20:32:27.918191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.918092    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a"} err="failed to get container status \"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a\": rpc error: code = NotFound desc = could not find container \"05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a\": container with ID starting with 05ecc9c67497fd7d92dd9d270a7d79c0b0c323aa6a3e81d574b85ed40fb58a2a not found: ID does not exist"
Nov 01 20:32:27.921109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:27.921089    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-c8tnz"]
Nov 01 20:32:28.431438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:28.431405    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" path="/var/lib/kubelet/pods/1c276b11-b7a7-4717-b385-9e1c04b1820a/volumes"
Nov 01 20:32:28.543182 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:28.543146    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:32:30.414392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414355    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"]
Nov 01 20:32:30.414940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414925    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:30.415002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414941    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:30.415002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414953    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:30.415002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414959    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:30.415002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414981    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c31b3de5-929e-41fb-a5e0-ce8941dabe10" containerName="revoke-keys"
Nov 01 20:32:30.415002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.414986    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31b3de5-929e-41fb-a5e0-ce8941dabe10" containerName="revoke-keys"
Nov 01 20:32:30.415156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.415082    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:30.415156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.415092    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:30.415156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.415104    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31b3de5-929e-41fb-a5e0-ce8941dabe10" containerName="revoke-keys"
Nov 01 20:32:30.415156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.415111    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:30.420237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.420207    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.424410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.424385    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:32:30.425057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.425035    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:32:30.425152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.425114    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:32:30.425221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.425189    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:32:30.430753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.430729    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"]
Nov 01 20:32:30.568787 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.568749    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.568958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.568901    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgk4\" (UniqueName: \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.569056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.569037    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.670853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.670737    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.670853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.670812    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgk4\" (UniqueName: \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.671083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.670885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.671499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.671469    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.673429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.673403    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.678559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.678531    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgk4\" (UniqueName: \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\") pod \"maas-api-revoke-sub-d1f10568-rzp8r\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:30.741417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:30.741383    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:31.027325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:31.027288    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"]
Nov 01 20:32:31.027735 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:32:31.027705    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f70690_1768_47af_a5ff_3a70f83096eb.slice/crio-5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f WatchSource:0}: Error finding container 5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f: Status 404 returned error can't find the container with id 5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f
Nov 01 20:32:31.910845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:31.909516    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r" event={"ID":"f3f70690-1768-47af-a5ff-3a70f83096eb","Type":"ContainerStarted","Data":"3c4bcf1312913bd76f74cb5fb151adad592b92445271d8cab34f03c418005796"}
Nov 01 20:32:31.910845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:31.909635    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r" event={"ID":"f3f70690-1768-47af-a5ff-3a70f83096eb","Type":"ContainerStarted","Data":"5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f"}
Nov 01 20:32:31.929910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:31.929760    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r" podStartSLOduration=1.9297419329999999 podStartE2EDuration="1.929741933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:31.926853576 +0000 UTC m=+2039.647932253" watchObservedRunningTime="2026-11-01 20:32:31.929741933 +0000 UTC m=+2039.650820608"
Nov 01 20:32:32.023277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:32.023209    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"]
Nov 01 20:32:32.027051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:32.027022    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f85707-mhwjh"]
Nov 01 20:32:32.428194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:32.428150    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cfc601-57ab-4780-83c3-65e20b7bf0dd" path="/var/lib/kubelet/pods/a7cfc601-57ab-4780-83c3-65e20b7bf0dd/volumes"
Nov 01 20:32:32.918103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:32.918065    5041 generic.go:356] "Generic (PLEG): container finished" podID="f3f70690-1768-47af-a5ff-3a70f83096eb" containerID="3c4bcf1312913bd76f74cb5fb151adad592b92445271d8cab34f03c418005796" exitCode=0
Nov 01 20:32:32.918550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:32.918141    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r" event={"ID":"f3f70690-1768-47af-a5ff-3a70f83096eb","Type":"ContainerDied","Data":"3c4bcf1312913bd76f74cb5fb151adad592b92445271d8cab34f03c418005796"}
Nov 01 20:32:34.094035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.094006    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:34.225125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.225045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\") pod \"f3f70690-1768-47af-a5ff-3a70f83096eb\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") "
Nov 01 20:32:34.225125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.225115    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\") pod \"f3f70690-1768-47af-a5ff-3a70f83096eb\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") "
Nov 01 20:32:34.225323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.225231    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\" (UniqueName: \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\") pod \"f3f70690-1768-47af-a5ff-3a70f83096eb\" (UID: \"f3f70690-1768-47af-a5ff-3a70f83096eb\") "
Nov 01 20:32:34.225574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.225512    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca" pod "f3f70690-1768-47af-a5ff-3a70f83096eb" (UID: "f3f70690-1768-47af-a5ff-3a70f83096eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:34.226011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.225987    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:34.227375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.227349    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca" pod "f3f70690-1768-47af-a5ff-3a70f83096eb" (UID: "f3f70690-1768-47af-a5ff-3a70f83096eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:34.227477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.227429    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4" pod "f3f70690-1768-47af-a5ff-3a70f83096eb" (UID: "f3f70690-1768-47af-a5ff-3a70f83096eb"). InnerVolumeSpecName "kube-api-access-wdgk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:34.327301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.327261    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f70690-1768-47af-a5ff-3a70f83096eb-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:34.327301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.327293    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdgk4\" (UniqueName: \"kubernetes.io/projected/f3f70690-1768-47af-a5ff-3a70f83096eb-kube-api-access-wdgk4\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:34.936830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.936774    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"
Nov 01 20:32:34.936830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.936814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r" event={"ID":"f3f70690-1768-47af-a5ff-3a70f83096eb","Type":"ContainerDied","Data":"5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f"}
Nov 01 20:32:34.937060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:34.936874    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bad450e5bc2f160d028ea9d96fb14302be53b32d236aded632a49e54b9e005f"
Nov 01 20:32:36.433013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:36.432985    5041 scope.go:122] "RemoveContainer" containerID="d71f8b4b660dcc97ccc4c1b4f79c9fa6309c52b366222903e5504ee7805e748a"
Nov 01 20:32:36.447151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:36.447127    5041 scope.go:122] "RemoveContainer" containerID="8def254543a77e028e6672b4c44ad54cf14eef1f2acf82f741f86a9fa50cc433"
Nov 01 20:32:37.386488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.386443    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main" containerID="cri-o://9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d" gracePeriod=45
Nov 01 20:32:37.697369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.697150    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:32:37.702112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.697869    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" containerID="cri-o://667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02" gracePeriod=30
Nov 01 20:32:37.721088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.721062    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:32:37.762800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.762735    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.762972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.762924    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\" (UniqueName: \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.763046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763018    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.763111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763058    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.763167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763112    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.763167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763124    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.763281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763172    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\") pod \"aa562803-46c5-4958-b192-133a03f0766f\" (UID: \"aa562803-46c5-4958-b192-133a03f0766f\") "
Nov 01 20:32:37.763622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763581    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.763622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.763603    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.764051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.764030    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.765548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.765511    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "kube-api-access-974h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:37.767064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.767029    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:37.767940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.767915    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs" pod "aa562803-46c5-4958-b192-133a03f0766f" (UID: "aa562803-46c5-4958-b192-133a03f0766f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:37.887340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.871482    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974h7\" (UniqueName: \"kubernetes.io/projected/aa562803-46c5-4958-b192-133a03f0766f-kube-api-access-974h7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.887340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.871521    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.887340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.871536    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa562803-46c5-4958-b192-133a03f0766f-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.887340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.871551    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.887340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.871566    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa562803-46c5-4958-b192-133a03f0766f-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:37.961111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.961022    5041 generic.go:356] "Generic (PLEG): container finished" podID="aa562803-46c5-4958-b192-133a03f0766f" containerID="9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d" exitCode=0
Nov 01 20:32:37.961111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.961053    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"
Nov 01 20:32:37.961111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.961090    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" event={"ID":"aa562803-46c5-4958-b192-133a03f0766f","Type":"ContainerDied","Data":"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"}
Nov 01 20:32:37.961354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.961136    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5" event={"ID":"aa562803-46c5-4958-b192-133a03f0766f","Type":"ContainerDied","Data":"df31f2b5b97d644a9b189cd17ad3589f3ee119700dbfefd51d03e813b28c2967"}
Nov 01 20:32:37.961354 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.961153    5041 scope.go:122] "RemoveContainer" containerID="9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"
Nov 01 20:32:37.973098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.973078    5041 scope.go:122] "RemoveContainer" containerID="9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"
Nov 01 20:32:37.973421 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:37.973397    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d\": container with ID starting with 9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d not found: ID does not exist" containerID="9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"
Nov 01 20:32:37.973514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.973427    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d"} err="failed to get container status \"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d\": rpc error: code = NotFound desc = could not find container \"9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d\": container with ID starting with 9af5dfbdde0a4639cbfa09e496633608281e291b743e1cc582e9924e8b13228d not found: ID does not exist"
Nov 01 20:32:37.986759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.986734    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:32:37.989994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:37.989969    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a6873b/simulator-w1-a6873b-kserve-789478ff89-vddl5"]
Nov 01 20:32:38.427584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:38.427549    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa562803-46c5-4958-b192-133a03f0766f" path="/var/lib/kubelet/pods/aa562803-46c5-4958-b192-133a03f0766f/volumes"
Nov 01 20:32:40.557952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:40.557913    5041 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:32:40.558388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:40.557991    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:32:42.990169 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:42.990131    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:32:43.002083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.002054    5041 generic.go:356] "Generic (PLEG): container finished" podID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerID="667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02" exitCode=0
Nov 01 20:32:43.002227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.002112    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" event={"ID":"fd11cbcc-3bf9-445f-8a97-3f79957e62b6","Type":"ContainerDied","Data":"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"}
Nov 01 20:32:43.002227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.002151    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6" event={"ID":"fd11cbcc-3bf9-445f-8a97-3f79957e62b6","Type":"ContainerDied","Data":"d55fc3302948593f0bee6ce6ad7835badc0574d68c2e3ce38e21b6c22343ceff"}
Nov 01 20:32:43.002227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.002176    5041 scope.go:122] "RemoveContainer" containerID="667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"
Nov 01 20:32:43.002365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.002285    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"
Nov 01 20:32:43.014476 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.014265    5041 scope.go:122] "RemoveContainer" containerID="667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"
Nov 01 20:32:43.015107 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:43.014644    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02\": container with ID starting with 667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02 not found: ID does not exist" containerID="667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"
Nov 01 20:32:43.015107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.014687    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02"} err="failed to get container status \"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02\": rpc error: code = NotFound desc = could not find container \"667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02\": container with ID starting with 667c1009e189bb443c8ad84290cb4b12e328a1261fedec4b9c61d6781e60ac02 not found: ID does not exist"
Nov 01 20:32:43.027047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.027021    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"]
Nov 01 20:32:43.032642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.032615    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4419b4ae-zp74v"]
Nov 01 20:32:43.155652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155613    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.155652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155655    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.155887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155691    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.155887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155720    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.155887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155754    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.155887 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155853    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.156112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155886    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.156112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155988    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:43.156112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.155978    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.156112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156075    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:43.156359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156195    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\") pod \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\" (UID: \"fd11cbcc-3bf9-445f-8a97-3f79957e62b6\") "
Nov 01 20:32:43.156359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156235    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:43.156485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156411    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:43.156626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:43.156773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156759    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.156877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156797    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.156877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156814    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.156877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156828    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.156877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.156843    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.158190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.158159    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:32:43.158305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.158193    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:32:43.158305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.158252    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:43.158719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.158703    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh" pod "fd11cbcc-3bf9-445f-8a97-3f79957e62b6" (UID: "fd11cbcc-3bf9-445f-8a97-3f79957e62b6"). InnerVolumeSpecName "kube-api-access-jckfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:43.257577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.257487    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jckfh\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-kube-api-access-jckfh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.257577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.257522    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.257577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.257533    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.257577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.257541    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11cbcc-3bf9-445f-8a97-3f79957e62b6-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:43.342046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.342006    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:32:43.345992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:43.345960    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-gc7f6"]
Nov 01 20:32:44.427664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:44.427634    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa3115e-595e-4468-84d0-adfe2fc0822f" path="/var/lib/kubelet/pods/efa3115e-595e-4468-84d0-adfe2fc0822f/volumes"
Nov 01 20:32:44.428312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:44.428294    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" path="/var/lib/kubelet/pods/fd11cbcc-3bf9-445f-8a97-3f79957e62b6/volumes"
Nov 01 20:32:52.476969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.476928    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"]
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477439    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477449    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3405bab-1ca0-433e-8c93-e3f0ad7e8b6d" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477462    5041 cpu_manager.go:414] "containerMap: removing container" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477467    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477488    5041 cpu_manager.go:414] "containerMap: removing container" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477493    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477506    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477511    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477518    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f3f70690-1768-47af-a5ff-3a70f83096eb" containerName="revoke-keys"
Nov 01 20:32:52.477586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477523    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f70690-1768-47af-a5ff-3a70f83096eb" containerName="revoke-keys"
Nov 01 20:32:52.478054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477618    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa562803-46c5-4958-b192-133a03f0766f" containerName="main"
Nov 01 20:32:52.478054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477629    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11cbcc-3bf9-445f-8a97-3f79957e62b6" containerName="istio-proxy"
Nov 01 20:32:52.478054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477639    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f70690-1768-47af-a5ff-3a70f83096eb" containerName="revoke-keys"
Nov 01 20:32:52.478054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.477650    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c276b11-b7a7-4717-b385-9e1c04b1820a" containerName="istio-proxy"
Nov 01 20:32:52.482725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.482706    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.486201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.486179    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:32:52.487307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.487286    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:32:52.487859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.487841    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:32:52.487982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.487916    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:32:52.489273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.489248    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"]
Nov 01 20:32:52.578623 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.578587    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.578845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.578730    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.578901 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.578876    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrzs\" (UniqueName: \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.679956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.679919    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.680132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.680016    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrzs\" (UniqueName: \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.680132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.680084    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.680625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.680605    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.682588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.682558    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.687821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.687796    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrzs\" (UniqueName: \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\") pod \"maas-api-revoke-sub-0f92b9b9-4vh9z\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:52.799620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:52.799593    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:53.085735 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:53.085707    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"]
Nov 01 20:32:53.087303 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:32:53.087280    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082a346f_bd6e_4f8e_8f72_b8c66f7c7e48.slice/crio-774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab WatchSource:0}: Error finding container 774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab: Status 404 returned error can't find the container with id 774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab
Nov 01 20:32:54.092664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:54.092622    5041 generic.go:356] "Generic (PLEG): container finished" podID="082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" containerID="51328d292ba8ebf3e97bdef7e8bf6adfaeedb285d5c072d86ad12e42fa841092" exitCode=0
Nov 01 20:32:54.093231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:54.092702    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z" event={"ID":"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48","Type":"ContainerDied","Data":"51328d292ba8ebf3e97bdef7e8bf6adfaeedb285d5c072d86ad12e42fa841092"}
Nov 01 20:32:54.093231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:54.092822    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z" event={"ID":"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48","Type":"ContainerStarted","Data":"774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab"}
Nov 01 20:32:55.241584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.241555    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:55.319891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.319855    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\" (UniqueName: \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\") pod \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") "
Nov 01 20:32:55.320083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.319901    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\") pod \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") "
Nov 01 20:32:55.320083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.319933    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\") pod \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\" (UID: \"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48\") "
Nov 01 20:32:55.320322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.320294    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca" pod "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" (UID: "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:32:55.320544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.320522    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:55.322244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.322212    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs" pod "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" (UID: "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48"). InnerVolumeSpecName "kube-api-access-lfrzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:32:55.322244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.322231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca" pod "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" (UID: "082a346f-bd6e-4f8e-8f72-b8c66f7c7e48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:32:55.421721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.421632    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrzs\" (UniqueName: \"kubernetes.io/projected/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-kube-api-access-lfrzs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:55.421721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:55.421666    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:32:56.108635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:56.108601    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"
Nov 01 20:32:56.108837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:56.108613    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z" event={"ID":"082a346f-bd6e-4f8e-8f72-b8c66f7c7e48","Type":"ContainerDied","Data":"774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab"}
Nov 01 20:32:56.108837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:32:56.108720    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="774bbf15d77beb6eccd12587ef75931437db98cae271cf03513ca39f3a8319ab"
Nov 01 20:32:58.542131 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:32:58.542094    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:33:01.707317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:01.707265    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:33:01.708725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:01.708469    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy" containerID="cri-o://f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b" gracePeriod=30
Nov 01 20:33:03.043126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.043091    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"]
Nov 01 20:33:03.043696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.043684    5041 cpu_manager.go:414] "containerMap: removing container" podUID="082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" containerName="revoke-keys"
Nov 01 20:33:03.043756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.043698    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" containerName="revoke-keys"
Nov 01 20:33:03.043842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.043818    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" containerName="revoke-keys"
Nov 01 20:33:03.047492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.047474    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.052020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.051992    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:33:03.052207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.052190    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:33:03.052391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.052378    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:33:03.052580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.052554    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:33:03.055203 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.055178    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"]
Nov 01 20:33:03.118335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.118289    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.118516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.118496    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjhc\" (UniqueName: \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.118571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.118548    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.219375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.219331    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.219585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.219481    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjhc\" (UniqueName: \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.219585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.219527    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.220165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.220139    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.222022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.222004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.228349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.228321    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjhc\" (UniqueName: \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\") pod \"maas-api-revoke-sub-9f05962d-ph6xd\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.294306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.294216    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:33:03.366031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.365993    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:03.672601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:03.672573    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"]
Nov 01 20:33:03.673972 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:33:03.673939    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f292bd_0695_48ec_8bc1_b13ee2afe80a.slice/crio-f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33 WatchSource:0}: Error finding container f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33: Status 404 returned error can't find the container with id f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33
Nov 01 20:33:04.171164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:04.171130    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd" event={"ID":"06f292bd-0695-48ec-8bc1-b13ee2afe80a","Type":"ContainerStarted","Data":"dac5080c2877c4fc4913dfc51a84a065b5457a2bb159fb650491f20ce5e260b3"}
Nov 01 20:33:04.171590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:04.171184    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd" event={"ID":"06f292bd-0695-48ec-8bc1-b13ee2afe80a","Type":"ContainerStarted","Data":"f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33"}
Nov 01 20:33:04.191461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:04.191404    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd" podStartSLOduration=1.19139097 podStartE2EDuration="1.19139097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:04.188459945 +0000 UTC m=+2071.909538622" watchObservedRunningTime="2026-11-01 20:33:04.19139097 +0000 UTC m=+2071.912469646"
Nov 01 20:33:05.179909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:05.179873    5041 generic.go:356] "Generic (PLEG): container finished" podID="06f292bd-0695-48ec-8bc1-b13ee2afe80a" containerID="dac5080c2877c4fc4913dfc51a84a065b5457a2bb159fb650491f20ce5e260b3" exitCode=0
Nov 01 20:33:05.180337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:05.179953    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd" event={"ID":"06f292bd-0695-48ec-8bc1-b13ee2afe80a","Type":"ContainerDied","Data":"dac5080c2877c4fc4913dfc51a84a065b5457a2bb159fb650491f20ce5e260b3"}
Nov 01 20:33:06.353225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.353196    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:06.481866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.481740    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\") pod \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") "
Nov 01 20:33:06.481866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.481849    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\" (UniqueName: \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\") pod \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") "
Nov 01 20:33:06.482098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.482015    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\") pod \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\" (UID: \"06f292bd-0695-48ec-8bc1-b13ee2afe80a\") "
Nov 01 20:33:06.482332 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.482304    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca" pod "06f292bd-0695-48ec-8bc1-b13ee2afe80a" (UID: "06f292bd-0695-48ec-8bc1-b13ee2afe80a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:06.482485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.482414    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:06.483961 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.483941    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca" pod "06f292bd-0695-48ec-8bc1-b13ee2afe80a" (UID: "06f292bd-0695-48ec-8bc1-b13ee2afe80a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:33:06.484052 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.483962    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc" pod "06f292bd-0695-48ec-8bc1-b13ee2afe80a" (UID: "06f292bd-0695-48ec-8bc1-b13ee2afe80a"). InnerVolumeSpecName "kube-api-access-ggjhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:06.584147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.584105    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f292bd-0695-48ec-8bc1-b13ee2afe80a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:06.584147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.584147    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjhc\" (UniqueName: \"kubernetes.io/projected/06f292bd-0695-48ec-8bc1-b13ee2afe80a-kube-api-access-ggjhc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:06.979823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:06.979762    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:33:07.089829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.089768    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.089870    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.089961    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090051    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090091    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090127    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090174    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090254    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\") pod \"65fa419f-07df-4189-ac37-a3b9217ae441\" (UID: \"65fa419f-07df-4189-ac37-a3b9217ae441\") "
Nov 01 20:33:07.090450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090294    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.090450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090208    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:07.090632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090514    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.090632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090525    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.090771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090740    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65fa419f-07df-4189-ac37-a3b9217ae441-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.090858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090790    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.090858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090807    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.090858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090822    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.090990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.090907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.092489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.092460    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:07.092720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.092699    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:07.092855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.092767    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:07.092855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.092811    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7" pod "65fa419f-07df-4189-ac37-a3b9217ae441" (UID: "65fa419f-07df-4189-ac37-a3b9217ae441"). InnerVolumeSpecName "kube-api-access-g7kl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:07.191511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.191473    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.191511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.191505    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.191745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.191519    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kl7\" (UniqueName: \"kubernetes.io/projected/65fa419f-07df-4189-ac37-a3b9217ae441-kube-api-access-g7kl7\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.191745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.191533    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65fa419f-07df-4189-ac37-a3b9217ae441-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.191745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.191545    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65fa419f-07df-4189-ac37-a3b9217ae441-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:07.195181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.195158    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"
Nov 01 20:33:07.195306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.195180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd" event={"ID":"06f292bd-0695-48ec-8bc1-b13ee2afe80a","Type":"ContainerDied","Data":"f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33"}
Nov 01 20:33:07.195306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.195225    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3bd520772fcae6b4b7a1c73ebc3445c2799ea4ab2f7f08f9620cd5bded21b33"
Nov 01 20:33:07.196630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.196602    5041 generic.go:356] "Generic (PLEG): container finished" podID="65fa419f-07df-4189-ac37-a3b9217ae441" containerID="f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b" exitCode=0
Nov 01 20:33:07.196743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.196642    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"
Nov 01 20:33:07.196743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.196666    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" event={"ID":"65fa419f-07df-4189-ac37-a3b9217ae441","Type":"ContainerDied","Data":"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"}
Nov 01 20:33:07.196743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.196715    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v" event={"ID":"65fa419f-07df-4189-ac37-a3b9217ae441","Type":"ContainerDied","Data":"15a1c14c2f1b5fb215263e77c876235a520d83511510e310c2f591ec56dd105a"}
Nov 01 20:33:07.197040 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.196748    5041 scope.go:122] "RemoveContainer" containerID="f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"
Nov 01 20:33:07.208373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.208352    5041 scope.go:122] "RemoveContainer" containerID="f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"
Nov 01 20:33:07.208693 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:07.208673    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b\": container with ID starting with f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b not found: ID does not exist" containerID="f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"
Nov 01 20:33:07.208744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.208703    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b"} err="failed to get container status \"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b\": rpc error: code = NotFound desc = could not find container \"f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b\": container with ID starting with f82e87c4556b597f6917fcda6e5dd84d90f11fe5198a0ae95cc1499bc1dbbd6b not found: ID does not exist"
Nov 01 20:33:07.228730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.228696    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:33:07.234235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:07.234211    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-qpz5v"]
Nov 01 20:33:08.294790 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:08.294736    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing" containerID="cri-o://1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9" gracePeriod=25
Nov 01 20:33:08.430816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:08.430737    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" path="/var/lib/kubelet/pods/65fa419f-07df-4189-ac37-a3b9217ae441/volumes"
Nov 01 20:33:15.169295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:15.169252    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:25.169486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:25.169451    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:33:28.546345 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:28.546204    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:33:28.591042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.591011    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:33:28.682277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.682239    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\") pod \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") "
Nov 01 20:33:28.682480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.682392    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\" (UniqueName: \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\") pod \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\" (UID: \"2f6f29ac-fbad-406f-8a3f-32a8ff40d427\") "
Nov 01 20:33:28.682646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.682620    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume" pod "2f6f29ac-fbad-406f-8a3f-32a8ff40d427" (UID: "2f6f29ac-fbad-406f-8a3f-32a8ff40d427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:28.682815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.682798    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:28.684577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.684548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm" pod "2f6f29ac-fbad-406f-8a3f-32a8ff40d427" (UID: "2f6f29ac-fbad-406f-8a3f-32a8ff40d427"). InnerVolumeSpecName "kube-api-access-mcmbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:28.783896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:28.783852    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmbm\" (UniqueName: \"kubernetes.io/projected/2f6f29ac-fbad-406f-8a3f-32a8ff40d427-kube-api-access-mcmbm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:29.361413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.361380    5041 generic.go:356] "Generic (PLEG): container finished" podID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerID="1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9" exitCode=0
Nov 01 20:33:29.361624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.361422    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"
Nov 01 20:33:29.361624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.361469    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" event={"ID":"2f6f29ac-fbad-406f-8a3f-32a8ff40d427","Type":"ContainerDied","Data":"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"}
Nov 01 20:33:29.361624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.361509    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd" event={"ID":"2f6f29ac-fbad-406f-8a3f-32a8ff40d427","Type":"ContainerDied","Data":"113c51b7593bb611a7e3d0d8f207d4e233fd2be70f717ea38977dd1ae66752c5"}
Nov 01 20:33:29.361624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.361536    5041 scope.go:122] "RemoveContainer" containerID="1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"
Nov 01 20:33:29.372420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.372396    5041 scope.go:122] "RemoveContainer" containerID="1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"
Nov 01 20:33:29.372718 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:29.372695    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9\": container with ID starting with 1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9 not found: ID does not exist" containerID="1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"
Nov 01 20:33:29.372833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.372726    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9"} err="failed to get container status \"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9\": rpc error: code = NotFound desc = could not find container \"1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9\": container with ID starting with 1c6451159d0076e6ee8b83a3dcd99673b9f7da36da5c374976c042363d65b1d9 not found: ID does not exist"
Nov 01 20:33:29.385341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.385303    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:33:29.390319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:29.390295    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0aa61b-6b6c668b4f-qjlfd"]
Nov 01 20:33:30.040242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:30.040206    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:33:30.040744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:30.040491    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" containerID="cri-o://a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be" gracePeriod=30
Nov 01 20:33:30.371301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:30.371217    5041 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:30.371301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:30.371275    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:30.431059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:30.431018    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" path="/var/lib/kubelet/pods/2f6f29ac-fbad-406f-8a3f-32a8ff40d427/volumes"
Nov 01 20:33:32.945938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:32.945907    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:33:32.947619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:32.947598    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:33:35.357573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.357539    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:33:35.412767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.412727    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:33:35.413081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.413030    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" containerID="cri-o://1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51" gracePeriod=30
Nov 01 20:33:35.437589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.437539    5041 generic.go:356] "Generic (PLEG): container finished" podID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerID="a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be" exitCode=0
Nov 01 20:33:35.437822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.437802    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerDied","Data":"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"}
Nov 01 20:33:35.437939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.437853    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8" event={"ID":"a40b3d08-2017-42dc-bfca-c5f7a9b04949","Type":"ContainerDied","Data":"dc2e26817ddd0d9fc6971993bbf6604be5be0d32a9e4e26a926b7881c438abd5"}
Nov 01 20:33:35.437939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.437881    5041 scope.go:122] "RemoveContainer" containerID="a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"
Nov 01 20:33:35.438067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.438046    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"
Nov 01 20:33:35.450404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.450382    5041 scope.go:122] "RemoveContainer" containerID="060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"
Nov 01 20:33:35.462180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.462154    5041 scope.go:122] "RemoveContainer" containerID="a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"
Nov 01 20:33:35.462486 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:35.462463    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be\": container with ID starting with a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be not found: ID does not exist" containerID="a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"
Nov 01 20:33:35.462637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.462496    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be"} err="failed to get container status \"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be\": rpc error: code = NotFound desc = could not find container \"a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be\": container with ID starting with a419309a29b3ebd816a2726d44dc3a27f680003b596a1f0f5fa36c00077732be not found: ID does not exist"
Nov 01 20:33:35.462637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.462517    5041 scope.go:122] "RemoveContainer" containerID="060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"
Nov 01 20:33:35.462884 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:35.462853    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666\": container with ID starting with 060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666 not found: ID does not exist" containerID="060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"
Nov 01 20:33:35.462980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.462893    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666"} err="failed to get container status \"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666\": rpc error: code = NotFound desc = could not find container \"060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666\": container with ID starting with 060d2a7a7f8069ea3211942a57499aa5b4c3bf7f03d0ba29187d5d9a350be666 not found: ID does not exist"
Nov 01 20:33:35.464377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464354    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464403    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464451    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464599 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464545    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464606    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:35.464657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464638    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464670    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464740    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464812    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.464922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464868    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\") pod \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\" (UID: \"a40b3d08-2017-42dc-bfca-c5f7a9b04949\") "
Nov 01 20:33:35.465027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464908    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:35.465027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.464980    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:35.465135 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465061    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:35.465305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465279    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:35.465378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465350    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.465378 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465368    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.465493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465379    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.465493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.465388    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.467063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.467035    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "kube-api-access-bqwqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:35.467154 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.467083    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:35.467344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.467323    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:35.467499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.467486    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy" pod "a40b3d08-2017-42dc-bfca-c5f7a9b04949" (UID: "a40b3d08-2017-42dc-bfca-c5f7a9b04949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:35.566513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.566322    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwqv\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-kube-api-access-bqwqv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.566513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.566391    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.566513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.566409    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.566513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.566424    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.566513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.566439    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a40b3d08-2017-42dc-bfca-c5f7a9b04949-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:35.762106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.762073    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:33:35.765645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:35.765619    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e83b4b-openshift-default-944f5867f-5ncx8"]
Nov 01 20:33:36.427576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:36.427545    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" path="/var/lib/kubelet/pods/a40b3d08-2017-42dc-bfca-c5f7a9b04949/volumes"
Nov 01 20:33:36.523496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:36.523463    5041 scope.go:122] "RemoveContainer" containerID="fc8eb43d1fc851a3488d521692879553dbba6b4def1c18032e2ac94ebf2e67d4"
Nov 01 20:33:36.533559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:36.533538    5041 scope.go:122] "RemoveContainer" containerID="16e41fe967ebd19989140f35bec8e6f2d52d64053ceeb9ad32e5bc65b6d7fb1d"
Nov 01 20:33:36.542362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:36.542344    5041 scope.go:122] "RemoveContainer" containerID="128a30801fc06eccdf609d02e1135e11d911654877bce159d1a178d2b5c9ce11"
Nov 01 20:33:36.551205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:36.551189    5041 scope.go:122] "RemoveContainer" containerID="ec47e0ad55513b4425d997520fcfdb9d462711c83d1b0b33465ffda795f94256"
Nov 01 20:33:37.026768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:37.026723    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"]
Nov 01 20:33:37.032198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:37.032161    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8a0f5b9-tvbsv"]
Nov 01 20:33:37.126402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:37.126370    5041 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:33:37.126575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:37.126425    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:33:38.021115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:38.021078    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"]
Nov 01 20:33:38.024387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:38.024362    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1ea27b9-phc5p"]
Nov 01 20:33:38.427190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:38.427155    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4be621-8e5f-4a64-ba35-c73af038444c" path="/var/lib/kubelet/pods/bc4be621-8e5f-4a64-ba35-c73af038444c/volumes"
Nov 01 20:33:38.427673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:38.427652    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb933f6-4d42-4351-89e1-f7ba574c74f8" path="/var/lib/kubelet/pods/ffb933f6-4d42-4351-89e1-f7ba574c74f8/volumes"
Nov 01 20:33:40.691848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.691756    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:33:40.727920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.727888    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728108 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.727955    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728208    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.728308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728288    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728343    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728392    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728434    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728490    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728595    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.728639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728611    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728708    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\") pod \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\" (UID: \"fb926a5d-b67e-4eb1-b634-aa07bfee99a5\") "
Nov 01 20:33:40.728919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728794    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.728919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.728887    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.729142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729122    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:40.729290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729272    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.729352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729296    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.729352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729312    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.729352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729326    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.729352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.729340    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.730480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.730458    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:40.730987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.730956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:40.731117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.731079    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:40.732196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.732169    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw" pod "fb926a5d-b67e-4eb1-b634-aa07bfee99a5" (UID: "fb926a5d-b67e-4eb1-b634-aa07bfee99a5"). InnerVolumeSpecName "kube-api-access-xkhrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:40.830648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.830613    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.830648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.830645    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.830648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.830657    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhrw\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-kube-api-access-xkhrw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:40.831090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:40.830668    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb926a5d-b67e-4eb1-b634-aa07bfee99a5-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:41.491497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.491465    5041 generic.go:356] "Generic (PLEG): container finished" podID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerID="1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51" exitCode=0
Nov 01 20:33:41.491717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.491535    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"
Nov 01 20:33:41.491717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.491550    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" event={"ID":"fb926a5d-b67e-4eb1-b634-aa07bfee99a5","Type":"ContainerDied","Data":"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"}
Nov 01 20:33:41.491717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.491602    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p" event={"ID":"fb926a5d-b67e-4eb1-b634-aa07bfee99a5","Type":"ContainerDied","Data":"382cd7f327ac5a4c297835bb45b08c2df5684e155e404d5a1146fc198932b9a3"}
Nov 01 20:33:41.491717 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.491623    5041 scope.go:122] "RemoveContainer" containerID="1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"
Nov 01 20:33:41.502371 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.502352    5041 scope.go:122] "RemoveContainer" containerID="1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"
Nov 01 20:33:41.502635 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:41.502617    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51\": container with ID starting with 1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51 not found: ID does not exist" containerID="1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"
Nov 01 20:33:41.502684 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.502644    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51"} err="failed to get container status \"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51\": rpc error: code = NotFound desc = could not find container \"1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51\": container with ID starting with 1087b52f2715f7f18c3c601ca43bc523781f8dea5d890c5a9f864be5a751ce51 not found: ID does not exist"
Nov 01 20:33:41.514655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.514624    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:33:41.519998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:41.519980    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-t979p"]
Nov 01 20:33:42.428069 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:42.428034    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" path="/var/lib/kubelet/pods/fb926a5d-b67e-4eb1-b634-aa07bfee99a5/volumes"
Nov 01 20:33:44.770399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:44.770314    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:33:44.770862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:44.770619    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy" containerID="cri-o://519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84" gracePeriod=30
Nov 01 20:33:50.024788 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.024737    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"]
Nov 01 20:33:50.026850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.026792    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:33:50.029035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.029015    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58b28677-fk654"]
Nov 01 20:33:50.141092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141057    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141111    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141150    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141206    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141270    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141338    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141384    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141429    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141470    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\") pod \"b7393207-eea7-4274-ae6e-edb2e1873ad3\" (UID: \"b7393207-eea7-4274-ae6e-edb2e1873ad3\") "
Nov 01 20:33:50.141710 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141548    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:33:50.141912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.141888    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7393207-eea7-4274-ae6e-edb2e1873ad3-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.142255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.142096    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:50.142255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.142117    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:50.142255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.142135    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:50.142255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.142220    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:50.143771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.143741    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:50.143907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.143856    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:33:50.143995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.143974    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "kube-api-access-n9t4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:33:50.144274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.144257    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo" pod "b7393207-eea7-4274-ae6e-edb2e1873ad3" (UID: "b7393207-eea7-4274-ae6e-edb2e1873ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:33:50.243503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243470    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243497    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243509    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243524    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243536    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7393207-eea7-4274-ae6e-edb2e1873ad3-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243547    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243556    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9t4l\" (UniqueName: \"kubernetes.io/projected/b7393207-eea7-4274-ae6e-edb2e1873ad3-kube-api-access-n9t4l\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.243699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.243566    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7393207-eea7-4274-ae6e-edb2e1873ad3-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:33:50.427772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.427695    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd79b4e-95ab-411f-a458-ccf14672d544" path="/var/lib/kubelet/pods/2cd79b4e-95ab-411f-a458-ccf14672d544/volumes"
Nov 01 20:33:50.554334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.554298    5041 generic.go:356] "Generic (PLEG): container finished" podID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerID="519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84" exitCode=0
Nov 01 20:33:50.554515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.554368    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"
Nov 01 20:33:50.554515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.554376    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerDied","Data":"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"}
Nov 01 20:33:50.554515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.554463    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4" event={"ID":"b7393207-eea7-4274-ae6e-edb2e1873ad3","Type":"ContainerDied","Data":"7e3b6aa252b9419467aa374c4aae18010c11a5e9c66f15e348946b0e80d77436"}
Nov 01 20:33:50.554515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.554494    5041 scope.go:122] "RemoveContainer" containerID="519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"
Nov 01 20:33:50.564924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.564905    5041 scope.go:122] "RemoveContainer" containerID="1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"
Nov 01 20:33:50.573482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.573456    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:33:50.575214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.575190    5041 scope.go:122] "RemoveContainer" containerID="519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"
Nov 01 20:33:50.575516 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:50.575489    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84\": container with ID starting with 519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84 not found: ID does not exist" containerID="519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"
Nov 01 20:33:50.575630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.575529    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84"} err="failed to get container status \"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84\": rpc error: code = NotFound desc = could not find container \"519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84\": container with ID starting with 519ffe3dcaff4be82570fafa04010eaca7b15ee1b7e94acb3b92960c8a261a84 not found: ID does not exist"
Nov 01 20:33:50.575630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.575554    5041 scope.go:122] "RemoveContainer" containerID="1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"
Nov 01 20:33:50.575858 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:50.575839    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27\": container with ID starting with 1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27 not found: ID does not exist" containerID="1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"
Nov 01 20:33:50.575934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.575866    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27"} err="failed to get container status \"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27\": rpc error: code = NotFound desc = could not find container \"1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27\": container with ID starting with 1e2bbe361c19a2618b56941520fe9e51081a8128c87769e6ecb4f129fdd11b27 not found: ID does not exist"
Nov 01 20:33:50.579675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:50.579653    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e38ef4-openshift-default-6b4dcb887c-6lcm4"]
Nov 01 20:33:52.427564 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:52.427527    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" path="/var/lib/kubelet/pods/b7393207-eea7-4274-ae6e-edb2e1873ad3/volumes"
Nov 01 20:33:54.019802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:54.019743    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"]
Nov 01 20:33:54.024323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:54.024292    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43b6628-89d7s"]
Nov 01 20:33:54.427148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:54.427118    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225629ac-15c9-4580-9c90-355fba578276" path="/var/lib/kubelet/pods/225629ac-15c9-4580-9c90-355fba578276/volumes"
Nov 01 20:33:58.218604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:58.218565    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:33:58.219150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:58.218857    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy" containerID="cri-o://d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994" gracePeriod=30
Nov 01 20:33:58.542651 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:33:58.542611    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:33:59.234795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:59.234735    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d8bc659cc-72rg2_40a08d27-42f8-48e2-8511-0032f24c6fd6/authorino/0.log"
Nov 01 20:33:59.423095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:59.423048    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/1.log"
Nov 01 20:33:59.593985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:59.593950    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9hf26_64f36934-c791-4d4a-b443-81e23ecb97ac/manager/0.log"
Nov 01 20:33:59.754417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:33:59.754390    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5fx26_74f17053-b4c9-40b0-8364-bbd0613232af/kuadrant-console-plugin/0.log"
Nov 01 20:34:00.506616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:00.506582    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rrr99_09f298b9-1369-4358-8649-2d3e06b15ba0/manager/0.log"
Nov 01 20:34:01.046170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:01.046140    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/0.log"
Nov 01 20:34:01.543001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:01.542968    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 20:34:02.764409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:02.764367    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-9tqcn_c296fd5a-f848-401e-990a-f1cdb60c5ccc/manager/0.log"
Nov 01 20:34:03.107339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.107307    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cj2xb_a902a934-ca05-4ef4-a015-35ca5504e645/manager/0.log"
Nov 01 20:34:03.488754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.488723    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:34:03.584023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.583990    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584081    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584151    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584243    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584351    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584387    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.584459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584434    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\") pod \"c43a2db1-474d-41b3-af31-61702130514b\" (UID: \"c43a2db1-474d-41b3-af31-61702130514b\") "
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584413    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584468    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584647    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584874    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.584967    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.585071    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.585094    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.585109    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c43a2db1-474d-41b3-af31-61702130514b-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.585122    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.585202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.585135    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.586441 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.586412    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:34:03.586549 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.586470    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "kube-api-access-8lgtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:03.586620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.586600    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:03.586709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.586692    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token" pod "c43a2db1-474d-41b3-af31-61702130514b" (UID: "c43a2db1-474d-41b3-af31-61702130514b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:03.639140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.639108    5041 generic.go:356] "Generic (PLEG): container finished" podID="c43a2db1-474d-41b3-af31-61702130514b" containerID="d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994" exitCode=0
Nov 01 20:34:03.639271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.639154    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"
Nov 01 20:34:03.639271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.639194    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" event={"ID":"c43a2db1-474d-41b3-af31-61702130514b","Type":"ContainerDied","Data":"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"}
Nov 01 20:34:03.639271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.639247    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2" event={"ID":"c43a2db1-474d-41b3-af31-61702130514b","Type":"ContainerDied","Data":"f18c915e7db7b0d72624a89bdec3cc58b89bcbf983c00804a0808ab15d7f5949"}
Nov 01 20:34:03.639401 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.639273    5041 scope.go:122] "RemoveContainer" containerID="d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"
Nov 01 20:34:03.650093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.650076    5041 scope.go:122] "RemoveContainer" containerID="d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"
Nov 01 20:34:03.650400 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:03.650380    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994\": container with ID starting with d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994 not found: ID does not exist" containerID="d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"
Nov 01 20:34:03.650461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.650408    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994"} err="failed to get container status \"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994\": rpc error: code = NotFound desc = could not find container \"d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994\": container with ID starting with d85d03a5b51df8a3662e62c169376bc4e564c185b0e9651ca3257e0e28be2994 not found: ID does not exist"
Nov 01 20:34:03.662223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.662197    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:34:03.668918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.668897    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-x4zz2"]
Nov 01 20:34:03.686699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.686668    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c43a2db1-474d-41b3-af31-61702130514b-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.686699 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.686700    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c43a2db1-474d-41b3-af31-61702130514b-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.686858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.686712    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lgtg\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-kube-api-access-8lgtg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:03.686858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:03.686722    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c43a2db1-474d-41b3-af31-61702130514b-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:04.428933 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:04.428901    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43a2db1-474d-41b3-af31-61702130514b" path="/var/lib/kubelet/pods/c43a2db1-474d-41b3-af31-61702130514b/volumes"
Nov 01 20:34:05.028424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:05.028370    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"]
Nov 01 20:34:05.031303 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:05.031277    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f2019b-7pcpp"]
Nov 01 20:34:06.427353 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:06.427316    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75187edf-276c-41ea-b61a-6609357ec1a0" path="/var/lib/kubelet/pods/75187edf-276c-41ea-b61a-6609357ec1a0/volumes"
Nov 01 20:34:15.029161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:15.029122    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"]
Nov 01 20:34:15.032869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:15.032834    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f35b6611-vbwbm"]
Nov 01 20:34:16.427588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:16.427549    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1530d04e-5598-4f5b-9aa6-e9f2754322e5" path="/var/lib/kubelet/pods/1530d04e-5598-4f5b-9aa6-e9f2754322e5/volumes"
Nov 01 20:34:25.021415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:25.021380    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"]
Nov 01 20:34:25.024120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:25.024093    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ed5f29-ltxwv"]
Nov 01 20:34:26.428214 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:26.428179    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13621076-3035-4dd2-950d-997e1e84340c" path="/var/lib/kubelet/pods/13621076-3035-4dd2-950d-997e1e84340c/volumes"
Nov 01 20:34:28.542163 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:28.542123    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:34:36.616973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:36.616934    5041 scope.go:122] "RemoveContainer" containerID="8d02b186e62a3ce6e82a563161f27c6c66bd139a21ec67f3dcd324ed27503fa8"
Nov 01 20:34:36.626728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:36.626703    5041 scope.go:122] "RemoveContainer" containerID="dee6de1f74e738d7ad18f938c82a029f183ecd98c41a5f3cc5a860b685ebd9b5"
Nov 01 20:34:36.635921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:36.635898    5041 scope.go:122] "RemoveContainer" containerID="d8997fdba9be15c5addf9b986bc5e99255dea7fa712e0fcfa835391dad6c8fde"
Nov 01 20:34:39.909647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.909612    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:34:39.910138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.909819    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy" containerID="cri-o://12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e" gracePeriod=30
Nov 01 20:34:39.912432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.912404    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:34:39.912667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.912632    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy" containerID="cri-o://4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369" gracePeriod=30
Nov 01 20:34:39.916006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.915975    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:34:39.916223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:39.916201    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy" containerID="cri-o://8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4" gracePeriod=30
Nov 01 20:34:45.196050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.195982    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:34:45.231741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.231626    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:34:45.242001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.241978    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:34:45.325443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325412    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.325646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325462    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.325646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325515    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.325646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325554    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.325646 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325583    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.325929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325710    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.325929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325727    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.325929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325759    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.325929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.325809    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326031    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.326162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326042    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.326162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326080    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.326162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326111    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326162 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326149    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326189    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326233    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326282    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326319    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326360    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\") pod \"542477d7-d721-481b-8972-6339c6c82be2\" (UID: \"542477d7-d721-481b-8972-6339c6c82be2\") "
Nov 01 20:34:45.326428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326397    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\") pod \"a10ac272-5665-4fdd-bc15-00763c2c49ca\" (UID: \"a10ac272-5665-4fdd-bc15-00763c2c49ca\") "
Nov 01 20:34:45.326760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326515    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:45.326980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326945    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542477d7-d721-481b-8972-6339c6c82be2-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.326980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326973    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.327109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.326988    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.327168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327110    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327379 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327356    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327484 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327437    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327512    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327651    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327729    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.327907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.327856    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:45.328444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.328422    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.328632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.328613    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.329322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.329303    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.329410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.329345    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "kube-api-access-2cscl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.329919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.329902    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo" pod "a10ac272-5665-4fdd-bc15-00763c2c49ca" (UID: "a10ac272-5665-4fdd-bc15-00763c2c49ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:34:45.330340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.330317    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:34:45.330530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.330510    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "kube-api-access-96l5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.330659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.330644    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy" pod "542477d7-d721-481b-8972-6339c6c82be2" (UID: "542477d7-d721-481b-8972-6339c6c82be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.428279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428245    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428317    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428375    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428425    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428454    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428497    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428648    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428680    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\") pod \"70b7612a-9446-4561-b1a5-059617ee2ff1\" (UID: \"70b7612a-9446-4561-b1a5-059617ee2ff1\") "
Nov 01 20:34:45.428886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428808    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.428886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.428692    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:34:45.429111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429008    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.429111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429075    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429136    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429194    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96l5g\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-kube-api-access-96l5g\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429209    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429218    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542477d7-d721-481b-8972-6339c6c82be2-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429227    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429236    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429245    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10ac272-5665-4fdd-bc15-00763c2c49ca-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429254    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542477d7-d721-481b-8972-6339c6c82be2-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429263    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429273    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429282    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429293    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70b7612a-9446-4561-b1a5-059617ee2ff1-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429302    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cscl\" (UniqueName: \"kubernetes.io/projected/a10ac272-5665-4fdd-bc15-00763c2c49ca-kube-api-access-2cscl\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429311    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429319    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10ac272-5665-4fdd-bc15-00763c2c49ca-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429331    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429345    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429359    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542477d7-d721-481b-8972-6339c6c82be2-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429372    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10ac272-5665-4fdd-bc15-00763c2c49ca-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.429521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.429380    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.431244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.431215    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:34:45.431244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.431222    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:34:45.431395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.431275    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "kube-api-access-jrwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.431395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.431293    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token" pod "70b7612a-9446-4561-b1a5-059617ee2ff1" (UID: "70b7612a-9446-4561-b1a5-059617ee2ff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:34:45.530039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.529942    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrwvs\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-kube-api-access-jrwvs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.530039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.529977    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.530039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.529990    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.530039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.530004    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70b7612a-9446-4561-b1a5-059617ee2ff1-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.530039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.530017    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70b7612a-9446-4561-b1a5-059617ee2ff1-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:34:45.915280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.915246    5041 generic.go:356] "Generic (PLEG): container finished" podID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerID="8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4" exitCode=0
Nov 01 20:34:45.915501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.915295    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"
Nov 01 20:34:45.915501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.915333    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" event={"ID":"70b7612a-9446-4561-b1a5-059617ee2ff1","Type":"ContainerDied","Data":"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"}
Nov 01 20:34:45.915501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.915399    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc" event={"ID":"70b7612a-9446-4561-b1a5-059617ee2ff1","Type":"ContainerDied","Data":"91f3e56aab6b3cd3a815cbddb183dd2984ff338eac64e24cb2a30f80097ad767"}
Nov 01 20:34:45.915501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.915429    5041 scope.go:122] "RemoveContainer" containerID="8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"
Nov 01 20:34:45.917250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.917226    5041 generic.go:356] "Generic (PLEG): container finished" podID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerID="12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e" exitCode=0
Nov 01 20:34:45.917362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.917276    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"
Nov 01 20:34:45.917362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.917294    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" event={"ID":"a10ac272-5665-4fdd-bc15-00763c2c49ca","Type":"ContainerDied","Data":"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"}
Nov 01 20:34:45.917362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.917342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln" event={"ID":"a10ac272-5665-4fdd-bc15-00763c2c49ca","Type":"ContainerDied","Data":"a072718928a950194f332da55b8769b2f320caff26e07c7f61fe335bc9bb90b1"}
Nov 01 20:34:45.919121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.919095    5041 generic.go:356] "Generic (PLEG): container finished" podID="542477d7-d721-481b-8972-6339c6c82be2" containerID="4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369" exitCode=0
Nov 01 20:34:45.919250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.919161    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" event={"ID":"542477d7-d721-481b-8972-6339c6c82be2","Type":"ContainerDied","Data":"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"}
Nov 01 20:34:45.919250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.919193    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77" event={"ID":"542477d7-d721-481b-8972-6339c6c82be2","Type":"ContainerDied","Data":"d9308d3758ca26413faa568403cd20dd244e24b376c661e3ffd66f719e737779"}
Nov 01 20:34:45.919384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.919278    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"
Nov 01 20:34:45.927313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.927296    5041 scope.go:122] "RemoveContainer" containerID="8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"
Nov 01 20:34:45.928131 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:45.928102    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4\": container with ID starting with 8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4 not found: ID does not exist" containerID="8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"
Nov 01 20:34:45.928233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.928143    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4"} err="failed to get container status \"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4\": rpc error: code = NotFound desc = could not find container \"8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4\": container with ID starting with 8052a6badeff0ef18bbbe2bb8a27c98a239a33bb2d08708031d5627f47cc50d4 not found: ID does not exist"
Nov 01 20:34:45.928233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.928175    5041 scope.go:122] "RemoveContainer" containerID="12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"
Nov 01 20:34:45.940020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.939996    5041 scope.go:122] "RemoveContainer" containerID="12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"
Nov 01 20:34:45.940376 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:45.940351    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e\": container with ID starting with 12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e not found: ID does not exist" containerID="12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"
Nov 01 20:34:45.940465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.940383    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e"} err="failed to get container status \"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e\": rpc error: code = NotFound desc = could not find container \"12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e\": container with ID starting with 12cf4a79191b6d462f015dc20a11bb144e08cf002fe1dde460aef8d8b2ab8f4e not found: ID does not exist"
Nov 01 20:34:45.940465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.940399    5041 scope.go:122] "RemoveContainer" containerID="4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"
Nov 01 20:34:45.949766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.949700    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:34:45.956693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.956642    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-8v5dc"]
Nov 01 20:34:45.963177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.962919    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:34:45.970572 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.970535    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-jmn77"]
Nov 01 20:34:45.970909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.970887    5041 scope.go:122] "RemoveContainer" containerID="4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"
Nov 01 20:34:45.971319 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:45.971295    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369\": container with ID starting with 4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369 not found: ID does not exist" containerID="4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"
Nov 01 20:34:45.971398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.971335    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369"} err="failed to get container status \"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369\": rpc error: code = NotFound desc = could not find container \"4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369\": container with ID starting with 4e91b7e3acafacb8782149bc24de49975bd1bc4731f637bd318bcdc56d795369 not found: ID does not exist"
Nov 01 20:34:45.990462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:45.990414    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:34:46.000590 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:46.000558    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6sln"]
Nov 01 20:34:46.429554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:46.429516    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542477d7-d721-481b-8972-6339c6c82be2" path="/var/lib/kubelet/pods/542477d7-d721-481b-8972-6339c6c82be2/volumes"
Nov 01 20:34:46.430238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:46.430205    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" path="/var/lib/kubelet/pods/70b7612a-9446-4561-b1a5-059617ee2ff1/volumes"
Nov 01 20:34:46.430871 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:34:46.430850    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" path="/var/lib/kubelet/pods/a10ac272-5665-4fdd-bc15-00763c2c49ca/volumes"
Nov 01 20:34:58.541371 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:34:58.541336    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:35:14.835867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:14.835764    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:35:14.836463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:14.835995    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" podUID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" containerName="manager" containerID="cri-o://d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb" gracePeriod=30
Nov 01 20:35:15.091625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.091563    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:35:15.110335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.110304    5041 generic.go:356] "Generic (PLEG): container finished" podID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" containerID="d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb" exitCode=0
Nov 01 20:35:15.110488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.110349    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-9tqcn"
Nov 01 20:35:15.110488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.110383    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" event={"ID":"c296fd5a-f848-401e-990a-f1cdb60c5ccc","Type":"ContainerDied","Data":"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"}
Nov 01 20:35:15.110488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.110432    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-9tqcn" event={"ID":"c296fd5a-f848-401e-990a-f1cdb60c5ccc","Type":"ContainerDied","Data":"bb32c631c5cd52492a679668aa40ac3ce029387841a01cf290435f8fd4163d5e"}
Nov 01 20:35:15.110488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.110454    5041 scope.go:122] "RemoveContainer" containerID="d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"
Nov 01 20:35:15.121094 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.121078    5041 scope.go:122] "RemoveContainer" containerID="d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"
Nov 01 20:35:15.121353 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:35:15.121334    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb\": container with ID starting with d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb not found: ID does not exist" containerID="d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"
Nov 01 20:35:15.121408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.121363    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb"} err="failed to get container status \"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb\": rpc error: code = NotFound desc = could not find container \"d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb\": container with ID starting with d22c1e39376eda68537bd9dda29bef6324e001ffd057b5bc63159bdd59fbe8bb not found: ID does not exist"
Nov 01 20:35:15.216305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.216275    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\" (UniqueName: \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\") pod \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") "
Nov 01 20:35:15.216500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.216313    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\") pod \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") "
Nov 01 20:35:15.216500 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.216369    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\") pod \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\" (UID: \"c296fd5a-f848-401e-990a-f1cdb60c5ccc\") "
Nov 01 20:35:15.218420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.218387    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs" pod "c296fd5a-f848-401e-990a-f1cdb60c5ccc" (UID: "c296fd5a-f848-401e-990a-f1cdb60c5ccc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:35:15.218538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.218444    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert" pod "c296fd5a-f848-401e-990a-f1cdb60c5ccc" (UID: "c296fd5a-f848-401e-990a-f1cdb60c5ccc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:35:15.218538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.218461    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh" pod "c296fd5a-f848-401e-990a-f1cdb60c5ccc" (UID: "c296fd5a-f848-401e-990a-f1cdb60c5ccc"). InnerVolumeSpecName "kube-api-access-78sbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:35:15.318053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.318012    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sbh\" (UniqueName: \"kubernetes.io/projected/c296fd5a-f848-401e-990a-f1cdb60c5ccc-kube-api-access-78sbh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:15.318053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.318047    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-metrics-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:15.318053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.318060    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c296fd5a-f848-401e-990a-f1cdb60c5ccc-webhook-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:15.434428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.434399    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:35:15.439957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:15.439934    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-9tqcn"]
Nov 01 20:35:16.103385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.103346    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:16.104024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104005    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" containerName="manager"
Nov 01 20:35:16.104083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104027    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" containerName="manager"
Nov 01 20:35:16.104083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104050    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy"
Nov 01 20:35:16.104083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104058    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy"
Nov 01 20:35:16.104083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104073    5041 cpu_manager.go:414] "containerMap: removing container" podUID="06f292bd-0695-48ec-8bc1-b13ee2afe80a" containerName="revoke-keys"
Nov 01 20:35:16.104083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104081    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f292bd-0695-48ec-8bc1-b13ee2afe80a" containerName="revoke-keys"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104097    5041 cpu_manager.go:414] "containerMap: removing container" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104105    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104122    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104129    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104146    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104154    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104168    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104176    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104194    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104202    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104225    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104232    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104242    5041 cpu_manager.go:414] "containerMap: removing container" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104249    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy"
Nov 01 20:35:16.104252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104267    5041 cpu_manager.go:414] "containerMap: removing container" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104285    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104295    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104302    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104430    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104444    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104456    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43a2db1-474d-41b3-af31-61702130514b" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104466    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fa419f-07df-4189-ac37-a3b9217ae441" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104476    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b7612a-9446-4561-b1a5-059617ee2ff1" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104485    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10ac272-5665-4fdd-bc15-00763c2c49ca" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104493    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" containerName="manager"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104503    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb926a5d-b67e-4eb1-b634-aa07bfee99a5" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104512    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f292bd-0695-48ec-8bc1-b13ee2afe80a" containerName="revoke-keys"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104519    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="542477d7-d721-481b-8972-6339c6c82be2" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104528    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:16.104746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.104539    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6f29ac-fbad-406f-8a3f-32a8ff40d427" containerName="payload-processing"
Nov 01 20:35:16.109679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.109654    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.116236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.116136    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 20:35:16.116503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.116404    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:16.116754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.116733    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 20:35:16.117207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.117189    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xqt97\""
Nov 01 20:35:16.117373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.117230    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 20:35:16.225836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.225772    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.226031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.225886    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fkl6\" (UniqueName: \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.226031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.225947    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.326638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.326591    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.326861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.326699    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.326861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.326765    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fkl6\" (UniqueName: \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.329143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.329112    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.329292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.329185    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.333926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.333904    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fkl6\" (UniqueName: \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\") pod \"maas-controller-7f997f45fb-p4fgb\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") " pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.427368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.427281    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c296fd5a-f848-401e-990a-f1cdb60c5ccc" path="/var/lib/kubelet/pods/c296fd5a-f848-401e-990a-f1cdb60c5ccc/volumes"
Nov 01 20:35:16.429338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.429313    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:16.706218 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.706194    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:16.709163 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:16.709136    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc677221_48d7_429a_a04b_f28d4a3ba81f.slice/crio-1c1ff5231c2637ec516346c3723043aa76bd9dfed2f2cbb079b4bada1902f6d5 WatchSource:0}: Error finding container 1c1ff5231c2637ec516346c3723043aa76bd9dfed2f2cbb079b4bada1902f6d5: Status 404 returned error can't find the container with id 1c1ff5231c2637ec516346c3723043aa76bd9dfed2f2cbb079b4bada1902f6d5
Nov 01 20:35:16.710910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:16.710892    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:35:17.126700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:17.126660    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" event={"ID":"cc677221-48d7-429a-a04b-f28d4a3ba81f","Type":"ContainerStarted","Data":"1c1ff5231c2637ec516346c3723043aa76bd9dfed2f2cbb079b4bada1902f6d5"}
Nov 01 20:35:18.141160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:18.141115    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" event={"ID":"cc677221-48d7-429a-a04b-f28d4a3ba81f","Type":"ContainerStarted","Data":"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"}
Nov 01 20:35:18.141607 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:18.141326    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:18.162902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:18.162847    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" podStartSLOduration=1.622001757 podStartE2EDuration="2.162833504s" totalImagesPullingTime="540.831747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:18.159549308 +0000 UTC m=+2205.880627984" watchObservedRunningTime="2026-11-01 20:35:18.162833504 +0000 UTC m=+2205.883912180"
Nov 01 20:35:22.881585 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.881547    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:35:22.882030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.882017    5041 cpu_manager.go:414] "containerMap: removing container" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:22.882089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.882031    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40b3d08-2017-42dc-bfca-c5f7a9b04949" containerName="istio-proxy"
Nov 01 20:35:22.882129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.882118    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7393207-eea7-4274-ae6e-edb2e1873ad3" containerName="istio-proxy"
Nov 01 20:35:22.885577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.885552    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.891651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.891628    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-1bfa0e-openshift-default-dockercfg-hbng5\""
Nov 01 20:35:22.895186 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.894942    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:35:22.993109 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993072    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993124    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993187    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwqd\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993277 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993210    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993294    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993322    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993407    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993455 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993447    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:22.993529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:22.993498    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095002    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095053    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwqd\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095077    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095112    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095222    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095288    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095343    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095389    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095445    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095479    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095518    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095842    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.095969 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.096026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.095974    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.097493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.097470    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.097842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.097823    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.103137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.103113    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.103249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.103166    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwqd\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.202596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.202513    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:23.480546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.480515    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:35:23.481467 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:23.481436    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c06eb5_1248_46b7_8c8f_ebb28eac050e.slice/crio-b0cd7b4525bff40fd26d670e32da4eccae3123352c282feee6da6025b016a51b WatchSource:0}: Error finding container b0cd7b4525bff40fd26d670e32da4eccae3123352c282feee6da6025b016a51b: Status 404 returned error can't find the container with id b0cd7b4525bff40fd26d670e32da4eccae3123352c282feee6da6025b016a51b
Nov 01 20:35:23.483657 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.483625    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:23.483759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.483685    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:23.483759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:23.483715    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:24.184851 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.184805    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" event={"ID":"83c06eb5-1248-46b7-8c8f-ebb28eac050e","Type":"ContainerStarted","Data":"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"}
Nov 01 20:35:24.185261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.184893    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" event={"ID":"83c06eb5-1248-46b7-8c8f-ebb28eac050e","Type":"ContainerStarted","Data":"b0cd7b4525bff40fd26d670e32da4eccae3123352c282feee6da6025b016a51b"}
Nov 01 20:35:24.203384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.203351    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:24.205340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.205307    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:24.205534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.205364    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:24.206873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:24.206828    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podStartSLOduration=2.2068164279999998 podStartE2EDuration="2.206816428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:24.204345756 +0000 UTC m=+2211.925424442" watchObservedRunningTime="2026-11-01 20:35:24.206816428 +0000 UTC m=+2211.927895103"
Nov 01 20:35:25.203446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:25.203411    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:25.203886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:25.203463    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:26.203576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:26.203545    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:26.204013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:26.203597    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:27.203167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:27.203132    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:27.203365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:27.203197    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:28.203248 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:28.203212    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:28.203757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:28.203265    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:28.541300 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:35:28.541268    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:35:29.167227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:29.167188    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:29.203310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:29.203277    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Nov 01 20:35:29.203799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:29.203342    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Nov 01 20:35:30.207949 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:30.207908    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:30.208363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:30.208168    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:30.208836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:30.208820    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:35:33.021216 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:33.021187    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"]
Nov 01 20:35:33.025667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:33.025644    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76900056-z72vg"]
Nov 01 20:35:34.428034 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:34.428001    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0b3905-ccc0-433d-8bf8-61db087fef5d" path="/var/lib/kubelet/pods/cb0b3905-ccc0-433d-8bf8-61db087fef5d/volumes"
Nov 01 20:35:36.688965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:36.688931    5041 scope.go:122] "RemoveContainer" containerID="b02de434d73b5d00505422f2165c7aca21e0255731531c5c75818daff7e56e69"
Nov 01 20:35:36.699329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:36.699304    5041 scope.go:122] "RemoveContainer" containerID="9975eb95e5e38501b8242583c8ce4dc0cd00215a13e9b2fcdf93dfb57acba594"
Nov 01 20:35:36.710707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:36.710676    5041 scope.go:122] "RemoveContainer" containerID="fa31e6c2833d03c49e8969f5cf5028467b03d7f1e04691ec554f44e1ca836c0e"
Nov 01 20:35:36.719139 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:36.719120    5041 scope.go:122] "RemoveContainer" containerID="423cf129323072d4c8b55eb1f5f5872fe727e399c70ae50a3e7058e9315a7bdf"
Nov 01 20:35:36.727221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:36.727202    5041 scope.go:122] "RemoveContainer" containerID="df6f0bb6d920f9c2edd4505fdcec052c5aca9ee29f30b476faccb9ce58ca6161"
Nov 01 20:35:37.289939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.289906    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"]
Nov 01 20:35:37.295240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.295222    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.298812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.298717    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:35:37.298935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.298895    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:35:37.300595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.300570    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"]
Nov 01 20:35:37.306554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.306515    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:35:37.306660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.306594    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:35:37.360067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.360032    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqkmk\" (UniqueName: \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.360236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.360104    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.360236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.360172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.461141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.461099    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.461370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.461152    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.461370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.461246    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqkmk\" (UniqueName: \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.462001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.461969    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.463839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.463804    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.468460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.468435    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqkmk\" (UniqueName: \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\") pod \"maas-api-revoke-sub-85237e45-vsjxd\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.610991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.610908    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:37.888511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:37.888480    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"]
Nov 01 20:35:37.889946 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:37.889918    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d54bea_2476_4c0b_a6c6_bd3812fb6998.slice/crio-c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c WatchSource:0}: Error finding container c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c: Status 404 returned error can't find the container with id c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c
Nov 01 20:35:38.270502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:38.270463    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd" event={"ID":"f6d54bea-2476-4c0b-a6c6-bd3812fb6998","Type":"ContainerStarted","Data":"7c26fcc098282ce2a3883c0d3bfc4f308879b957c48c509a9f9c625a900e3e14"}
Nov 01 20:35:38.270706 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:38.270519    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd" event={"ID":"f6d54bea-2476-4c0b-a6c6-bd3812fb6998","Type":"ContainerStarted","Data":"c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c"}
Nov 01 20:35:38.284962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:38.284911    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd" podStartSLOduration=1.28489828 podStartE2EDuration="1.28489828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:38.284084718 +0000 UTC m=+2226.005163418" watchObservedRunningTime="2026-11-01 20:35:38.28489828 +0000 UTC m=+2226.005976972"
Nov 01 20:35:39.278107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:39.278067    5041 generic.go:356] "Generic (PLEG): container finished" podID="f6d54bea-2476-4c0b-a6c6-bd3812fb6998" containerID="7c26fcc098282ce2a3883c0d3bfc4f308879b957c48c509a9f9c625a900e3e14" exitCode=0
Nov 01 20:35:39.278576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:39.278118    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd" event={"ID":"f6d54bea-2476-4c0b-a6c6-bd3812fb6998","Type":"ContainerDied","Data":"7c26fcc098282ce2a3883c0d3bfc4f308879b957c48c509a9f9c625a900e3e14"}
Nov 01 20:35:39.993028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:39.992988    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"]
Nov 01 20:35:39.997176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:39.997148    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.004703 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.004675    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"]
Nov 01 20:35:40.008965 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.008942    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.013198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.013175    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"]
Nov 01 20:35:40.028715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.028685    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"]
Nov 01 20:35:40.087946 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.087908    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.087961    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.087989    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kb8\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-kube-api-access-x6kb8\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088049    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9873f918-5b69-4ddd-b8f5-e27afea58726-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088125    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088166    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088185    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088238    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088262    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088350    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088381    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088460    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088513    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088562    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088622    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.088722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088661    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88d8\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-kube-api-access-c88d8\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088689    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.088910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.088727    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.189866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189825    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.189866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189870    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.189866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189889    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189935    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189961    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.189994    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190034    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190103    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190157    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88d8\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-kube-api-access-c88d8\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190199    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190223    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190268    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190305    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190331    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kb8\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-kube-api-access-x6kb8\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190358    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9873f918-5b69-4ddd-b8f5-e27afea58726-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190405    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190418    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.190738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190711    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.191192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190326    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.191192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191020    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.191192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191036    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.191373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191241    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.191373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191314    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.191373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.190329    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.191552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191533    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9873f918-5b69-4ddd-b8f5-e27afea58726-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.191831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.191804    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.192905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.192872    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.193100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.193083    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.193449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.193431    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.193697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.193676    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.207796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.207750    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.207948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.207927    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kb8\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-kube-api-access-x6kb8\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.208098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.208076    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9873f918-5b69-4ddd-b8f5-e27afea58726-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-qzbgz\" (UID: \"9873f918-5b69-4ddd-b8f5-e27afea58726\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.209007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.208989    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88d8\" (UniqueName: \"kubernetes.io/projected/3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd-kube-api-access-c88d8\") pod \"maas-default-gateway-openshift-default-fd4b99879-p6vnd\" (UID: \"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.313087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.313057    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:40.328533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.328503    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:40.450559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.450532    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:40.495632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.495602    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\") pod \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") "
Nov 01 20:35:40.495822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.495701    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\") pod \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") "
Nov 01 20:35:40.495822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.495766    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\" (UniqueName: \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\") pod \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\" (UID: \"f6d54bea-2476-4c0b-a6c6-bd3812fb6998\") "
Nov 01 20:35:40.496086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.496013    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca" pod "f6d54bea-2476-4c0b-a6c6-bd3812fb6998" (UID: "f6d54bea-2476-4c0b-a6c6-bd3812fb6998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:35:40.497764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.497738    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca" pod "f6d54bea-2476-4c0b-a6c6-bd3812fb6998" (UID: "f6d54bea-2476-4c0b-a6c6-bd3812fb6998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:35:40.497924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.497902    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk" pod "f6d54bea-2476-4c0b-a6c6-bd3812fb6998" (UID: "f6d54bea-2476-4c0b-a6c6-bd3812fb6998"). InnerVolumeSpecName "kube-api-access-hqkmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:35:40.596937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.596862    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:40.596937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.596899    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:40.596937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.596915    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqkmk\" (UniqueName: \"kubernetes.io/projected/f6d54bea-2476-4c0b-a6c6-bd3812fb6998-kube-api-access-hqkmk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:40.615967 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.615914    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"]
Nov 01 20:35:40.618294 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:40.618273    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3662c1fb_5ab7_4dbd_b7d4_fd8d434c43cd.slice/crio-c9e7c8ab3dc22654cfc8160e944bf41ba22ff5d1d63750ac28df9e07151318bb WatchSource:0}: Error finding container c9e7c8ab3dc22654cfc8160e944bf41ba22ff5d1d63750ac28df9e07151318bb: Status 404 returned error can't find the container with id c9e7c8ab3dc22654cfc8160e944bf41ba22ff5d1d63750ac28df9e07151318bb
Nov 01 20:35:40.620604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.620574    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:40.620673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.620633    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:40.620673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.620662    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:40.633103 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.633064    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"]
Nov 01 20:35:40.634022 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:40.633992    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9873f918_5b69_4ddd_b8f5_e27afea58726.slice/crio-b9fe1224381ee8596490becd49047dff6c58a97e2f4e2a3a2a24caf3b2c79bfe WatchSource:0}: Error finding container b9fe1224381ee8596490becd49047dff6c58a97e2f4e2a3a2a24caf3b2c79bfe: Status 404 returned error can't find the container with id b9fe1224381ee8596490becd49047dff6c58a97e2f4e2a3a2a24caf3b2c79bfe
Nov 01 20:35:40.636202 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.636169    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:40.636292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.636233    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:40.636292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:40.636264    5041 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 01 20:35:41.297648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.297620    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"
Nov 01 20:35:41.297863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.297616    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd" event={"ID":"f6d54bea-2476-4c0b-a6c6-bd3812fb6998","Type":"ContainerDied","Data":"c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c"}
Nov 01 20:35:41.297863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.297750    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61950cdc2d2d14417cc7884b00038f963717186a453513205e76b58796af98c"
Nov 01 20:35:41.299237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.299215    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" event={"ID":"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd","Type":"ContainerStarted","Data":"fd8534aaad64a194953c87c9fe19364593c84cf2f75f5b1d58cea47fac5537bd"}
Nov 01 20:35:41.299367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.299250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" event={"ID":"3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd","Type":"ContainerStarted","Data":"c9e7c8ab3dc22654cfc8160e944bf41ba22ff5d1d63750ac28df9e07151318bb"}
Nov 01 20:35:41.302017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.301995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" event={"ID":"9873f918-5b69-4ddd-b8f5-e27afea58726","Type":"ContainerStarted","Data":"db2be20267b9eca404a48192f653be53d264a0fc8bad060ae90010830914e33d"}
Nov 01 20:35:41.302132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.302031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" event={"ID":"9873f918-5b69-4ddd-b8f5-e27afea58726","Type":"ContainerStarted","Data":"b9fe1224381ee8596490becd49047dff6c58a97e2f4e2a3a2a24caf3b2c79bfe"}
Nov 01 20:35:41.313637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.313618    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:41.315408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.315375    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:41.315523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.315434    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:41.319864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.319824    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podStartSLOduration=2.319811669 podStartE2EDuration="2.319811669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:41.318191299 +0000 UTC m=+2229.039269975" watchObservedRunningTime="2026-11-01 20:35:41.319811669 +0000 UTC m=+2229.040890345"
Nov 01 20:35:41.332316 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.332294    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:41.333953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.333933    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:41.334050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.333971    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:41.356799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:41.354993    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podStartSLOduration=2.354970348 podStartE2EDuration="2.354970348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:41.340504255 +0000 UTC m=+2229.061582943" watchObservedRunningTime="2026-11-01 20:35:41.354970348 +0000 UTC m=+2229.076049025"
Nov 01 20:35:42.313938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:42.313908    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:42.314341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:42.313955    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:42.332055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:42.332026    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:42.332210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:42.332076    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:43.314418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:43.314376    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:43.314923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:43.314432    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:43.332693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:43.332660    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:43.332859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:43.332717    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:44.021608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.021569    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"]
Nov 01 20:35:44.024133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.024109    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2e02845-p9xw4"]
Nov 01 20:35:44.313803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.313673    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:44.313803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.313737    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:44.332956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.332919    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:44.333346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.332969    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:44.427700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:44.427660    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12392895-6c60-46a2-b646-09f9efe18b4a" path="/var/lib/kubelet/pods/12392895-6c60-46a2-b646-09f9efe18b4a/volumes"
Nov 01 20:35:45.314411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:45.314379    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:45.314600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:45.314432    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:45.332445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:45.332416    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:45.332620 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:45.332464    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:46.313998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:46.313969    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:46.316509 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:46.314020    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:46.332131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:46.332106    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:46.332290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:46.332149    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:47.314587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:47.314547    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body=
Nov 01 20:35:47.315134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:47.314600    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused"
Nov 01 20:35:47.332555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:47.332515    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 01 20:35:47.332714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:47.332569    5041 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 01 20:35:48.836842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:48.836802    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:48.837840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:48.837753    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerName="manager" containerID="cri-o://bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1" gracePeriod=30
Nov 01 20:35:49.148472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.148370    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.252:8081/readyz\": dial tcp 10.132.0.252:8081: connect: connection refused"
Nov 01 20:35:49.173180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.173143    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:49.175122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.174693    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:49.176531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.176026    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz"
Nov 01 20:35:49.206307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.206266    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:49.207195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.207139    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:49.208296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.208264    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd"
Nov 01 20:35:49.313562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.313534    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:49.358683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.358645    5041 generic.go:356] "Generic (PLEG): container finished" podID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerID="bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1" exitCode=0
Nov 01 20:35:49.358922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.358708    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-p4fgb"
Nov 01 20:35:49.358922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.358752    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" event={"ID":"cc677221-48d7-429a-a04b-f28d4a3ba81f","Type":"ContainerDied","Data":"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"}
Nov 01 20:35:49.358922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.358844    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-p4fgb" event={"ID":"cc677221-48d7-429a-a04b-f28d4a3ba81f","Type":"ContainerDied","Data":"1c1ff5231c2637ec516346c3723043aa76bd9dfed2f2cbb079b4bada1902f6d5"}
Nov 01 20:35:49.358922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.358874    5041 scope.go:122] "RemoveContainer" containerID="bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"
Nov 01 20:35:49.369714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.369693    5041 scope.go:122] "RemoveContainer" containerID="bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"
Nov 01 20:35:49.370065 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:35:49.370040    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1\": container with ID starting with bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1 not found: ID does not exist" containerID="bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"
Nov 01 20:35:49.370155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.370077    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1"} err="failed to get container status \"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1\": rpc error: code = NotFound desc = could not find container \"bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1\": container with ID starting with bce0894ae973469cee908d20bdb5710c688a2600f6ad5f231ce5ccf4072b68f1 not found: ID does not exist"
Nov 01 20:35:49.466614 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.466532    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\") pod \"cc677221-48d7-429a-a04b-f28d4a3ba81f\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") "
Nov 01 20:35:49.466827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.466634    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\") pod \"cc677221-48d7-429a-a04b-f28d4a3ba81f\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") "
Nov 01 20:35:49.466827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.466715    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\" (UniqueName: \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\") pod \"cc677221-48d7-429a-a04b-f28d4a3ba81f\" (UID: \"cc677221-48d7-429a-a04b-f28d4a3ba81f\") "
Nov 01 20:35:49.468845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.468819    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert" pod "cc677221-48d7-429a-a04b-f28d4a3ba81f" (UID: "cc677221-48d7-429a-a04b-f28d4a3ba81f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:35:49.468956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.468875    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6" pod "cc677221-48d7-429a-a04b-f28d4a3ba81f" (UID: "cc677221-48d7-429a-a04b-f28d4a3ba81f"). InnerVolumeSpecName "kube-api-access-8fkl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:35:49.469006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.468956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs" pod "cc677221-48d7-429a-a04b-f28d4a3ba81f" (UID: "cc677221-48d7-429a-a04b-f28d4a3ba81f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:35:49.568085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.568040    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fkl6\" (UniqueName: \"kubernetes.io/projected/cc677221-48d7-429a-a04b-f28d4a3ba81f-kube-api-access-8fkl6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:49.568085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.568080    5041 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-webhook-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:49.568085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.568089    5041 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc677221-48d7-429a-a04b-f28d4a3ba81f-metrics-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:35:49.681866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.681796    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:49.685140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:49.685117    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-p4fgb"]
Nov 01 20:35:50.427383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.427350    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" path="/var/lib/kubelet/pods/cc677221-48d7-429a-a04b-f28d4a3ba81f/volumes"
Nov 01 20:35:50.925489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.925455    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f997f45fb-2rd9z"]
Nov 01 20:35:50.925937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.925923    5041 cpu_manager.go:414] "containerMap: removing container" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerName="manager"
Nov 01 20:35:50.926007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.925937    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerName="manager"
Nov 01 20:35:50.926007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.925947    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f6d54bea-2476-4c0b-a6c6-bd3812fb6998" containerName="revoke-keys"
Nov 01 20:35:50.926007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.925952    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d54bea-2476-4c0b-a6c6-bd3812fb6998" containerName="revoke-keys"
Nov 01 20:35:50.926111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.926054    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc677221-48d7-429a-a04b-f28d4a3ba81f" containerName="manager"
Nov 01 20:35:50.926111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.926064    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d54bea-2476-4c0b-a6c6-bd3812fb6998" containerName="revoke-keys"
Nov 01 20:35:50.930409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.930390    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:50.935552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.935531    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 20:35:50.937504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.937318    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xqt97\""
Nov 01 20:35:50.937504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.937426    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 20:35:50.937504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.937479    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 20:35:50.939459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:50.939439    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-2rd9z"]
Nov 01 20:35:51.083768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.083735    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-metrics-certs\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.083975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.083905    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqshn\" (UniqueName: \"kubernetes.io/projected/97308fcb-f04e-43de-8e97-c3144625c0ef-kube-api-access-qqshn\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.083975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.083951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-webhook-cert\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.185288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.185198    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-metrics-certs\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.185477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.185327    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqshn\" (UniqueName: \"kubernetes.io/projected/97308fcb-f04e-43de-8e97-c3144625c0ef-kube-api-access-qqshn\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.185477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.185386    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-webhook-cert\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.187767 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.187742    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-metrics-certs\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.187922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.187814    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97308fcb-f04e-43de-8e97-c3144625c0ef-webhook-cert\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.192810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.192761    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqshn\" (UniqueName: \"kubernetes.io/projected/97308fcb-f04e-43de-8e97-c3144625c0ef-kube-api-access-qqshn\") pod \"maas-controller-7f997f45fb-2rd9z\" (UID: \"97308fcb-f04e-43de-8e97-c3144625c0ef\") " pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.246221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.246188    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:51.522442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:51.522416    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f997f45fb-2rd9z"]
Nov 01 20:35:51.524049 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:35:51.524022    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97308fcb_f04e_43de_8e97_c3144625c0ef.slice/crio-fcb3db37b1ce315cd77719ae4e9322fe3a5972ca58961e1c7f1900bf7d42e433 WatchSource:0}: Error finding container fcb3db37b1ce315cd77719ae4e9322fe3a5972ca58961e1c7f1900bf7d42e433: Status 404 returned error can't find the container with id fcb3db37b1ce315cd77719ae4e9322fe3a5972ca58961e1c7f1900bf7d42e433
Nov 01 20:35:52.378839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:52.378766    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-2rd9z" event={"ID":"97308fcb-f04e-43de-8e97-c3144625c0ef","Type":"ContainerStarted","Data":"9c40d9a0b1741c46799037abf61b113cc43e47fbf164a486149610b72d635c56"}
Nov 01 20:35:52.379048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:52.378865    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:35:52.379048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:52.378883    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f997f45fb-2rd9z" event={"ID":"97308fcb-f04e-43de-8e97-c3144625c0ef","Type":"ContainerStarted","Data":"fcb3db37b1ce315cd77719ae4e9322fe3a5972ca58961e1c7f1900bf7d42e433"}
Nov 01 20:35:52.398547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:52.398499    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f997f45fb-2rd9z" podStartSLOduration=2.019137086 podStartE2EDuration="2.398485511s" totalImagesPullingTime="379.348425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:35:52.394892904 +0000 UTC m=+2240.115971581" watchObservedRunningTime="2026-11-01 20:35:52.398485511 +0000 UTC m=+2240.119564186"
Nov 01 20:35:54.023292 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:54.023251    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"]
Nov 01 20:35:54.025885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:54.025858    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa8ab57-tj882"]
Nov 01 20:35:54.426916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:35:54.426883    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27" path="/var/lib/kubelet/pods/8b3ef7ef-f64a-4f8c-a888-1f21c3e35d27/volumes"
Nov 01 20:35:58.541927 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:35:58.541888    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:36:03.387854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:03.387821    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f997f45fb-2rd9z"
Nov 01 20:36:05.019129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:05.019091    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"]
Nov 01 20:36:05.023802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:05.023762    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6de262a6-7jqz5"]
Nov 01 20:36:06.427634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:06.427603    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8888f1ef-dde2-4066-8889-29b4f469d157" path="/var/lib/kubelet/pods/8888f1ef-dde2-4066-8889-29b4f469d157/volumes"
Nov 01 20:36:28.541381 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:36:28.541335    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:36:33.024404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:33.024365    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"]
Nov 01 20:36:33.026899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:33.026875    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91086483-jwtgb"]
Nov 01 20:36:34.427983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:34.427951    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f63d6f-26f0-4586-a9f7-64d722b5f021" path="/var/lib/kubelet/pods/63f63d6f-26f0-4586-a9f7-64d722b5f021/volumes"
Nov 01 20:36:36.781463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:36.781429    5041 scope.go:122] "RemoveContainer" containerID="37bcd88c88a959aad366dded7ed244f6eee228cb2ab63f85b0750de526ea2c72"
Nov 01 20:36:45.020765 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:45.020684    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"]
Nov 01 20:36:45.024649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:45.024624    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-621b3900-mtxbt"]
Nov 01 20:36:46.428457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:36:46.428428    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04575f7d-d2ed-4642-90a4-fb38871a3d16" path="/var/lib/kubelet/pods/04575f7d-d2ed-4642-90a4-fb38871a3d16/volumes"
Nov 01 20:36:58.545933 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:36:58.545891    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:37:21.023217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:21.023177    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"]
Nov 01 20:37:21.025756 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:21.025727    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edb3d6e8-vltxn"]
Nov 01 20:37:22.428192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:22.428158    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31b3de5-929e-41fb-a5e0-ce8941dabe10" path="/var/lib/kubelet/pods/c31b3de5-929e-41fb-a5e0-ce8941dabe10/volumes"
Nov 01 20:37:28.541233 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:37:28.541194    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:37:34.022884 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:34.022847    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"]
Nov 01 20:37:34.026160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:34.026129    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f10568-rzp8r"]
Nov 01 20:37:34.429429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:34.429394    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f70690-1768-47af-a5ff-3a70f83096eb" path="/var/lib/kubelet/pods/f3f70690-1768-47af-a5ff-3a70f83096eb/volumes"
Nov 01 20:37:36.827141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:36.827103    5041 scope.go:122] "RemoveContainer" containerID="f4168f7cc2beb295556a27df8a2c5647c74d428ecaac02530d80bc9f49af6de7"
Nov 01 20:37:36.836701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:36.836679    5041 scope.go:122] "RemoveContainer" containerID="81f90c5b82060d8c80f660d69eefe0af5f6394d372b8c0695a08bd7a7a5954a6"
Nov 01 20:37:36.851238 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:36.851205    5041 scope.go:122] "RemoveContainer" containerID="874b1c714c9da6b9aecad9f87a88a1b54c79cc1c91676bc15747ddf725148363"
Nov 01 20:37:36.860754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:36.860734    5041 scope.go:122] "RemoveContainer" containerID="235b84bb38582c57fdb66a7ed64e8380b728655a6459f1dfd1d30c1d4e31deaf"
Nov 01 20:37:48.356802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:48.352900    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:37:48.356802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:48.352996    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:37:56.026430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:56.026379    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"]
Nov 01 20:37:56.029181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:56.029152    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f92b9b9-4vh9z"]
Nov 01 20:37:56.427200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:37:56.427166    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="082a346f-bd6e-4f8e-8f72-b8c66f7c7e48" path="/var/lib/kubelet/pods/082a346f-bd6e-4f8e-8f72-b8c66f7c7e48/volumes"
Nov 01 20:37:58.541085 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:37:58.541045    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:38:07.022608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:07.022572    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"]
Nov 01 20:38:07.024738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:07.024715    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f05962d-ph6xd"]
Nov 01 20:38:08.427120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:08.427085    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f292bd-0695-48ec-8bc1-b13ee2afe80a" path="/var/lib/kubelet/pods/06f292bd-0695-48ec-8bc1-b13ee2afe80a/volumes"
Nov 01 20:38:19.931272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:19.931035    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:38:19.932194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:19.932139    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerName="discovery" containerID="cri-o://e14ca4e23a101115c90470df8ac9d2b411a825802de1e2fd5323b1f5b32930ec" gracePeriod=30
Nov 01 20:38:20.246973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.246880    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:20.247227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.247195    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:20.260001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.259887    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:20.260001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.259937    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:20.548811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.542408    5041 generic.go:356] "Generic (PLEG): container finished" podID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerID="e14ca4e23a101115c90470df8ac9d2b411a825802de1e2fd5323b1f5b32930ec" exitCode=0
Nov 01 20:38:20.548811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.542591    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" event={"ID":"4b286be8-2d7d-41cd-89b1-ef48aaec78fc","Type":"ContainerDied","Data":"e14ca4e23a101115c90470df8ac9d2b411a825802de1e2fd5323b1f5b32930ec"}
Nov 01 20:38:20.719188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.719141    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:38:20.735090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735298    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735397    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735502    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735626 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735613    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735794    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.735955 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.735939    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\") pod \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\" (UID: \"4b286be8-2d7d-41cd-89b1-ef48aaec78fc\") "
Nov 01 20:38:20.736184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.736161    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:20.736503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.736483    5041 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-ca-configmap\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.738269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.738238    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:20.738386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.738355    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:38:20.738548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.738529    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:20.740072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.740040    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "kube-api-access-5wwbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:20.740251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.740230    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:20.741080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.741054    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig" pod "4b286be8-2d7d-41cd-89b1-ef48aaec78fc" (UID: "4b286be8-2d7d-41cd-89b1-ef48aaec78fc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837508    5041 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-csr-dns-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837538    5041 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-kubeconfig\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837547    5041 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-cacerts\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837555    5041 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-local-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837564    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwbs\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-kube-api-access-5wwbs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:20.837872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:20.837575    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b286be8-2d7d-41cd-89b1-ef48aaec78fc-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:21.549460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:21.549424    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" event={"ID":"4b286be8-2d7d-41cd-89b1-ef48aaec78fc","Type":"ContainerDied","Data":"bf37d90e63a508e7a9ff283209565fbaba0124b75c0fea70f0cd34aa6887f355"}
Nov 01 20:38:21.549460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:21.549448    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"
Nov 01 20:38:21.550056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:21.549490    5041 scope.go:122] "RemoveContainer" containerID="e14ca4e23a101115c90470df8ac9d2b411a825802de1e2fd5323b1f5b32930ec"
Nov 01 20:38:21.573896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:21.573866    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:38:21.579639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:21.579614    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7"]
Nov 01 20:38:22.427200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.427157    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" path="/var/lib/kubelet/pods/4b286be8-2d7d-41cd-89b1-ef48aaec78fc/volumes"
Nov 01 20:38:22.784930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.784897    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"]
Nov 01 20:38:22.785616 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.785594    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerName="discovery"
Nov 01 20:38:22.785742 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.785619    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerName="discovery"
Nov 01 20:38:22.785850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.785752    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerName="discovery"
Nov 01 20:38:22.790425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.790402    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.795919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.795897    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"]
Nov 01 20:38:22.797890 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.797868    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:38:22.797987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.797880    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:38:22.797987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.797964    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:38:22.798198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.798178    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:38:22.860016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.859983    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.860224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.860022    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nf48\" (UniqueName: \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.860224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.860047    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.961659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.961620    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.961868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.961666    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nf48\" (UniqueName: \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.961868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.961691    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.962363 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.962340    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.964120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.964096    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:22.969104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:22.969086    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nf48\" (UniqueName: \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\") pod \"maas-api-revoke-sub-117cdddd-f6v2t\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:23.108115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:23.108034    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:23.382386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:23.382355    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"]
Nov 01 20:38:23.383928 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:38:23.383899    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44c282e_3c2a_4e94_b296_b02aed12446c.slice/crio-67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad WatchSource:0}: Error finding container 67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad: Status 404 returned error can't find the container with id 67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad
Nov 01 20:38:23.563487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:23.563447    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t" event={"ID":"d44c282e-3c2a-4e94-b296-b02aed12446c","Type":"ContainerStarted","Data":"67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad"}
Nov 01 20:38:24.571358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:24.571326    5041 generic.go:356] "Generic (PLEG): container finished" podID="d44c282e-3c2a-4e94-b296-b02aed12446c" containerID="41e7a9330a2f4bf9e7c457aa7f5c6927ee949e392bb7772e80f30949f76efa63" exitCode=0
Nov 01 20:38:24.571768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:24.571390    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t" event={"ID":"d44c282e-3c2a-4e94-b296-b02aed12446c","Type":"ContainerDied","Data":"41e7a9330a2f4bf9e7c457aa7f5c6927ee949e392bb7772e80f30949f76efa63"}
Nov 01 20:38:25.680023 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.679983    5041 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b2vf7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:38:25.680428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.680053    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2vf7" podUID="4b286be8-2d7d-41cd-89b1-ef48aaec78fc" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.112:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:38:25.732575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.732549    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:25.789907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.789873    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\" (UniqueName: \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\") pod \"d44c282e-3c2a-4e94-b296-b02aed12446c\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") "
Nov 01 20:38:25.790083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.790063    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\") pod \"d44c282e-3c2a-4e94-b296-b02aed12446c\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") "
Nov 01 20:38:25.790168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.790149    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\") pod \"d44c282e-3c2a-4e94-b296-b02aed12446c\" (UID: \"d44c282e-3c2a-4e94-b296-b02aed12446c\") "
Nov 01 20:38:25.790457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.790430    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca" pod "d44c282e-3c2a-4e94-b296-b02aed12446c" (UID: "d44c282e-3c2a-4e94-b296-b02aed12446c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:25.790566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.790532    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:25.791902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.791874    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48" pod "d44c282e-3c2a-4e94-b296-b02aed12446c" (UID: "d44c282e-3c2a-4e94-b296-b02aed12446c"). InnerVolumeSpecName "kube-api-access-5nf48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:25.792312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.792285    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca" pod "d44c282e-3c2a-4e94-b296-b02aed12446c" (UID: "d44c282e-3c2a-4e94-b296-b02aed12446c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:25.891498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.891424    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44c282e-3c2a-4e94-b296-b02aed12446c-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:25.891498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:25.891452    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nf48\" (UniqueName: \"kubernetes.io/projected/d44c282e-3c2a-4e94-b296-b02aed12446c-kube-api-access-5nf48\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:26.584724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:26.584682    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t" event={"ID":"d44c282e-3c2a-4e94-b296-b02aed12446c","Type":"ContainerDied","Data":"67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad"}
Nov 01 20:38:26.584914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:26.584736    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67951ab029826cee819f6d15f72d755557ce15b703593348317d4170d844cfad"
Nov 01 20:38:26.584914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:26.584704    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"
Nov 01 20:38:28.541588 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:38:28.541551    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:38:32.992183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.992151    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"]
Nov 01 20:38:32.992842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.992788    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:38:32.993001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.992858    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d44c282e-3c2a-4e94-b296-b02aed12446c" containerName="revoke-keys"
Nov 01 20:38:32.993001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.992877    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44c282e-3c2a-4e94-b296-b02aed12446c" containerName="revoke-keys"
Nov 01 20:38:32.993126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.993023    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44c282e-3c2a-4e94-b296-b02aed12446c" containerName="revoke-keys"
Nov 01 20:38:32.996723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.996701    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:32.998754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:32.998733    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:38:33.001349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.001328    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:38:33.001586 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.001464    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:38:33.001713 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.001546    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:38:33.001827 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.001600    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:38:33.005165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.005145    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"]
Nov 01 20:38:33.067124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.067091    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhlt\" (UniqueName: \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.067301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.067138    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.067364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.067335    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.168895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.168860    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhlt\" (UniqueName: \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.169083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.168913    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.169083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.169011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.169686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.169659    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.171467 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.171443    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.177181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.177159    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhlt\" (UniqueName: \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\") pod \"maas-api-revoke-sub-ef9206f2-6grg4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.316042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.316011    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:33.598881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.598857    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"]
Nov 01 20:38:33.601104 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:38:33.601075    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d37387_559b_4619_a269_51bad362e9a4.slice/crio-4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336 WatchSource:0}: Error finding container 4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336: Status 404 returned error can't find the container with id 4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336
Nov 01 20:38:33.635615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:33.635580    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4" event={"ID":"84d37387-559b-4619-a269-51bad362e9a4","Type":"ContainerStarted","Data":"4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336"}
Nov 01 20:38:34.643080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:34.643047    5041 generic.go:356] "Generic (PLEG): container finished" podID="84d37387-559b-4619-a269-51bad362e9a4" containerID="f736e45726db359f244c1b145c1712f086fa8d5f99dabf2deb606c13550936d4" exitCode=0
Nov 01 20:38:34.643478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:34.643127    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4" event={"ID":"84d37387-559b-4619-a269-51bad362e9a4","Type":"ContainerDied","Data":"f736e45726db359f244c1b145c1712f086fa8d5f99dabf2deb606c13550936d4"}
Nov 01 20:38:35.845853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.845547    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:35.912352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.911606    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\") pod \"84d37387-559b-4619-a269-51bad362e9a4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") "
Nov 01 20:38:35.912352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.911708    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\") pod \"84d37387-559b-4619-a269-51bad362e9a4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") "
Nov 01 20:38:35.912352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.911823    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\" (UniqueName: \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\") pod \"84d37387-559b-4619-a269-51bad362e9a4\" (UID: \"84d37387-559b-4619-a269-51bad362e9a4\") "
Nov 01 20:38:35.914031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.913721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca" pod "84d37387-559b-4619-a269-51bad362e9a4" (UID: "84d37387-559b-4619-a269-51bad362e9a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:35.914734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.914705    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca" pod "84d37387-559b-4619-a269-51bad362e9a4" (UID: "84d37387-559b-4619-a269-51bad362e9a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:35.922000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:35.921959    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt" pod "84d37387-559b-4619-a269-51bad362e9a4" (UID: "84d37387-559b-4619-a269-51bad362e9a4"). InnerVolumeSpecName "kube-api-access-8zhlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:36.013695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.013588    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d37387-559b-4619-a269-51bad362e9a4-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:36.013695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.013639    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhlt\" (UniqueName: \"kubernetes.io/projected/84d37387-559b-4619-a269-51bad362e9a4-kube-api-access-8zhlt\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:36.013695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.013656    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d37387-559b-4619-a269-51bad362e9a4-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:36.691501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.683995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4" event={"ID":"84d37387-559b-4619-a269-51bad362e9a4","Type":"ContainerDied","Data":"4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336"}
Nov 01 20:38:36.691501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.684068    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e7f80171df857442ad186f381c8ae46110725a09aa879e1af9d04a623989336"
Nov 01 20:38:36.691501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.684139    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"
Nov 01 20:38:36.921667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.919294    5041 scope.go:122] "RemoveContainer" containerID="3c4bcf1312913bd76f74cb5fb151adad592b92445271d8cab34f03c418005796"
Nov 01 20:38:36.937072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.932651    5041 scope.go:122] "RemoveContainer" containerID="52d8e80bafcd3e7162a47cce86dff0065fbf1bdd7c0d93ea3fcede14be41b344"
Nov 01 20:38:36.948136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:36.946734    5041 scope.go:122] "RemoveContainer" containerID="beb53d4a8b7dc9387ee9418a856796961112e8f0cd72a0f729eb498fc572dd0b"
Nov 01 20:38:43.293987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.293945    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"]
Nov 01 20:38:43.294543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.294525    5041 cpu_manager.go:414] "containerMap: removing container" podUID="84d37387-559b-4619-a269-51bad362e9a4" containerName="revoke-keys"
Nov 01 20:38:43.294600 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.294544    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d37387-559b-4619-a269-51bad362e9a4" containerName="revoke-keys"
Nov 01 20:38:43.294675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.294665    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d37387-559b-4619-a269-51bad362e9a4" containerName="revoke-keys"
Nov 01 20:38:43.299592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.299573    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.304817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.302988    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:38:43.304817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.304132    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:38:43.304817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.304146    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:38:43.304817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.304444    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"]
Nov 01 20:38:43.304817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.304620    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:38:43.399247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.399216    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.399449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.399258    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.399449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.399413    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.500046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.500007    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.500046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.500052    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.500351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.500107    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.500848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.500827    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.502438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.502421    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.508634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.508610    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\") pod \"maas-api-revoke-sub-0fd24d96-ktf2v\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.617193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.617106    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:43.893309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:43.893283    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"]
Nov 01 20:38:43.895458 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:38:43.895430    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ce6510_f58c_42fa_8fbd_19dcc202b3c4.slice/crio-c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554 WatchSource:0}: Error finding container c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554: Status 404 returned error can't find the container with id c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554
Nov 01 20:38:44.757298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:44.757271    5041 generic.go:356] "Generic (PLEG): container finished" podID="e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" containerID="d2ca1174460a54f71ae6a672223c8c9f76ce580f31572e186265a7f35c73329d" exitCode=0
Nov 01 20:38:44.757615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:44.757348    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v" event={"ID":"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4","Type":"ContainerDied","Data":"d2ca1174460a54f71ae6a672223c8c9f76ce580f31572e186265a7f35c73329d"}
Nov 01 20:38:44.757615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:44.757394    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v" event={"ID":"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4","Type":"ContainerStarted","Data":"c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554"}
Nov 01 20:38:45.921149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:45.921113    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:46.026664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.026575    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\") pod \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") "
Nov 01 20:38:46.026664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.026627    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\") pod \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") "
Nov 01 20:38:46.026958 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.026858    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\") pod \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\" (UID: \"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4\") "
Nov 01 20:38:46.027266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.027244    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca" pod "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" (UID: "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:46.028787 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.028754    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca" pod "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" (UID: "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:46.028889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.028820    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27" pod "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" (UID: "e9ce6510-f58c-42fa-8fbd-19dcc202b3c4"). InnerVolumeSpecName "kube-api-access-wxd27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:46.128373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.128343    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.128373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.128371    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.128373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.128382    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxd27\" (UniqueName: \"kubernetes.io/projected/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4-kube-api-access-wxd27\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:46.769978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.769882    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"
Nov 01 20:38:46.769978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.769908    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v" event={"ID":"e9ce6510-f58c-42fa-8fbd-19dcc202b3c4","Type":"ContainerDied","Data":"c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554"}
Nov 01 20:38:46.770189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:46.769983    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48bdc7a9739b5db4f432c9be8c582a06c2dc57628f39cea92e92f58a9271554"
Nov 01 20:38:53.510464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.510427    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"]
Nov 01 20:38:53.510996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.510916    5041 cpu_manager.go:414] "containerMap: removing container" podUID="e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" containerName="revoke-keys"
Nov 01 20:38:53.510996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.510928    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" containerName="revoke-keys"
Nov 01 20:38:53.511100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.511016    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" containerName="revoke-keys"
Nov 01 20:38:53.514498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.514480    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.519342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.519316    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:38:53.519634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.519611    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:38:53.522172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.522147    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"]
Nov 01 20:38:53.590364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.590330    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:38:53.590364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.590352    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:38:53.604930 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.604899    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.605086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.604963    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.605137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.605093    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4lv\" (UniqueName: \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.706310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.706270    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4lv\" (UniqueName: \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.706515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.706371    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.706515 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.706415    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.707072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.707047    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.708802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.708763    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.713900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.713880    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4lv\" (UniqueName: \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\") pod \"maas-api-revoke-sub-18977da6-9587j\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:53.830743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:53.830703    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:54.113050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:54.113026    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"]
Nov 01 20:38:54.114181 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:38:54.114151    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea404f3_41d9_4955_b8bb_385190f6ac3a.slice/crio-a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47 WatchSource:0}: Error finding container a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47: Status 404 returned error can't find the container with id a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47
Nov 01 20:38:54.818381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:54.818342    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j" event={"ID":"0ea404f3-41d9-4955-b8bb-385190f6ac3a","Type":"ContainerStarted","Data":"5d964d332597112698135779b89239ca803beed0b820dcc626635fba659f7483"}
Nov 01 20:38:54.818919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:54.818405    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j" event={"ID":"0ea404f3-41d9-4955-b8bb-385190f6ac3a","Type":"ContainerStarted","Data":"a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47"}
Nov 01 20:38:54.848364 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:54.847395    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j" podStartSLOduration=1.847363157 podStartE2EDuration="1.847363157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:38:54.844461339 +0000 UTC m=+2422.565540026" watchObservedRunningTime="2026-11-01 20:38:54.847363157 +0000 UTC m=+2422.568441835"
Nov 01 20:38:55.830446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:55.829869    5041 generic.go:356] "Generic (PLEG): container finished" podID="0ea404f3-41d9-4955-b8bb-385190f6ac3a" containerID="5d964d332597112698135779b89239ca803beed0b820dcc626635fba659f7483" exitCode=0
Nov 01 20:38:55.830446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:55.830081    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j" event={"ID":"0ea404f3-41d9-4955-b8bb-385190f6ac3a","Type":"ContainerDied","Data":"5d964d332597112698135779b89239ca803beed0b820dcc626635fba659f7483"}
Nov 01 20:38:57.319847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.316626    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:57.334136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.334102    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\") pod \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") "
Nov 01 20:38:57.336128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.334190    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\") pod \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") "
Nov 01 20:38:57.336128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.334523    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca" pod "0ea404f3-41d9-4955-b8bb-385190f6ac3a" (UID: "0ea404f3-41d9-4955-b8bb-385190f6ac3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:38:57.338187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.338162    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca" pod "0ea404f3-41d9-4955-b8bb-385190f6ac3a" (UID: "0ea404f3-41d9-4955-b8bb-385190f6ac3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:38:57.434876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.434770    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\" (UniqueName: \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\") pod \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\" (UID: \"0ea404f3-41d9-4955-b8bb-385190f6ac3a\") "
Nov 01 20:38:57.435227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.435202    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:57.435319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.435234    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ea404f3-41d9-4955-b8bb-385190f6ac3a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:57.437155 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.437121    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv" pod "0ea404f3-41d9-4955-b8bb-385190f6ac3a" (UID: "0ea404f3-41d9-4955-b8bb-385190f6ac3a"). InnerVolumeSpecName "kube-api-access-nc4lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:38:57.535879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.535840    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4lv\" (UniqueName: \"kubernetes.io/projected/0ea404f3-41d9-4955-b8bb-385190f6ac3a-kube-api-access-nc4lv\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:38:57.843984 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.843953    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"
Nov 01 20:38:57.844168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.843946    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j" event={"ID":"0ea404f3-41d9-4955-b8bb-385190f6ac3a","Type":"ContainerDied","Data":"a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47"}
Nov 01 20:38:57.844168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:38:57.844081    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e4c64d68ae29860f821803e200b994e6975e3bd7b7ebc19c862a2c78e8cb47"
Nov 01 20:38:58.541749 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:38:58.541715    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:39:03.786878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.786841    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"]
Nov 01 20:39:03.787507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.787484    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0ea404f3-41d9-4955-b8bb-385190f6ac3a" containerName="revoke-keys"
Nov 01 20:39:03.787618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.787510    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea404f3-41d9-4955-b8bb-385190f6ac3a" containerName="revoke-keys"
Nov 01 20:39:03.787683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.787674    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea404f3-41d9-4955-b8bb-385190f6ac3a" containerName="revoke-keys"
Nov 01 20:39:03.791018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.790998    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.795809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.795751    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:03.795809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.795764    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:03.795809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.795758    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:03.796076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.795950    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:03.799589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.799431    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"]
Nov 01 20:39:03.897276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.897209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrj6\" (UniqueName: \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.897462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.897333    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.897462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.897425    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.998670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.998634    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrj6\" (UniqueName: \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.998919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.998703    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.998919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.998738    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:03.999449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:03.999424    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:04.001120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.001097    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:04.006833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.006805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrj6\" (UniqueName: \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\") pod \"maas-api-revoke-sub-4e1d2fca-r777l\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:04.106563 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.106480    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:04.385115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.385081    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"]
Nov 01 20:39:04.386312 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:04.386285    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c449f39_e24e_43ef_9357_e5e850a643c3.slice/crio-3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e WatchSource:0}: Error finding container 3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e: Status 404 returned error can't find the container with id 3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e
Nov 01 20:39:04.895067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.895026    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l" event={"ID":"8c449f39-e24e-43ef-9357-e5e850a643c3","Type":"ContainerStarted","Data":"0c2d0446adfa7e13779a122a32fc51b342de3734097766ad9d1ae3e3410ce68e"}
Nov 01 20:39:04.895552 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.895084    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l" event={"ID":"8c449f39-e24e-43ef-9357-e5e850a643c3","Type":"ContainerStarted","Data":"3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e"}
Nov 01 20:39:04.913180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:04.913093    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l" podStartSLOduration=1.9130795680000001 podStartE2EDuration="1.913079568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:04.90955539 +0000 UTC m=+2432.630634067" watchObservedRunningTime="2026-11-01 20:39:04.913079568 +0000 UTC m=+2432.634158244"
Nov 01 20:39:05.905069 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:05.905025    5041 generic.go:356] "Generic (PLEG): container finished" podID="8c449f39-e24e-43ef-9357-e5e850a643c3" containerID="0c2d0446adfa7e13779a122a32fc51b342de3734097766ad9d1ae3e3410ce68e" exitCode=0
Nov 01 20:39:05.905525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:05.905113    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l" event={"ID":"8c449f39-e24e-43ef-9357-e5e850a643c3","Type":"ContainerDied","Data":"0c2d0446adfa7e13779a122a32fc51b342de3734097766ad9d1ae3e3410ce68e"}
Nov 01 20:39:07.292125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.289381    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:07.380820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.380089    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\") pod \"8c449f39-e24e-43ef-9357-e5e850a643c3\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") "
Nov 01 20:39:07.380820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.380232    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\" (UniqueName: \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\") pod \"8c449f39-e24e-43ef-9357-e5e850a643c3\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") "
Nov 01 20:39:07.380820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.380275    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\") pod \"8c449f39-e24e-43ef-9357-e5e850a643c3\" (UID: \"8c449f39-e24e-43ef-9357-e5e850a643c3\") "
Nov 01 20:39:07.392991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.392956    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca" pod "8c449f39-e24e-43ef-9357-e5e850a643c3" (UID: "8c449f39-e24e-43ef-9357-e5e850a643c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:07.424463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.419916    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6" pod "8c449f39-e24e-43ef-9357-e5e850a643c3" (UID: "8c449f39-e24e-43ef-9357-e5e850a643c3"). InnerVolumeSpecName "kube-api-access-jkrj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:07.424463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.423857    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca" pod "8c449f39-e24e-43ef-9357-e5e850a643c3" (UID: "8c449f39-e24e-43ef-9357-e5e850a643c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:07.490001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.483664    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:07.490001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.489847    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c449f39-e24e-43ef-9357-e5e850a643c3-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:07.490001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.489893    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrj6\" (UniqueName: \"kubernetes.io/projected/8c449f39-e24e-43ef-9357-e5e850a643c3-kube-api-access-jkrj6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:07.933882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.933853    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"
Nov 01 20:39:07.933882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.933866    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l" event={"ID":"8c449f39-e24e-43ef-9357-e5e850a643c3","Type":"ContainerDied","Data":"3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e"}
Nov 01 20:39:07.934126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:07.933919    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b6d1995b261effc651b32821f2594b029c14ec68d71fe65ca85ddbc3355ac2e"
Nov 01 20:39:13.998922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:13.998890    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"]
Nov 01 20:39:13.999356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:13.999343    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8c449f39-e24e-43ef-9357-e5e850a643c3" containerName="revoke-keys"
Nov 01 20:39:13.999407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:13.999356    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c449f39-e24e-43ef-9357-e5e850a643c3" containerName="revoke-keys"
Nov 01 20:39:13.999460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:13.999451    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c449f39-e24e-43ef-9357-e5e850a643c3" containerName="revoke-keys"
Nov 01 20:39:14.002769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.002749    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.006228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.006205    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:14.009295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.009272    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"]
Nov 01 20:39:14.010728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.010705    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:14.010910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.010757    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:14.106516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.106486    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:14.162125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.162087    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.162297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.162131    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.162297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.162160    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr968\" (UniqueName: \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.263659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.263578    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.263659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.263616    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.263659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.263638    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr968\" (UniqueName: \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.264341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.264318    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.266076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.266053    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.271153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.271133    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr968\" (UniqueName: \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\") pod \"maas-api-revoke-sub-54c431e2-clrz4\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.325102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.325064    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:14.599672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.599647    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"]
Nov 01 20:39:14.602456 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:14.602427    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5af236d_1015_4e9e_bb8b_8ecb3366e2f9.slice/crio-4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04 WatchSource:0}: Error finding container 4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04: Status 404 returned error can't find the container with id 4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04
Nov 01 20:39:14.977339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.977298    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4" event={"ID":"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9","Type":"ContainerStarted","Data":"3fdb13241b62ba27148dd495a23fec7938a6153cd285b9395104cd9ce178fe5c"}
Nov 01 20:39:14.977560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.977363    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4" event={"ID":"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9","Type":"ContainerStarted","Data":"4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04"}
Nov 01 20:39:14.994987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:14.994931    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4" podStartSLOduration=1.9949181660000002 podStartE2EDuration="1.994918166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:14.992620382 +0000 UTC m=+2442.713699060" watchObservedRunningTime="2026-11-01 20:39:14.994918166 +0000 UTC m=+2442.715996842"
Nov 01 20:39:15.988814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:15.988747    5041 generic.go:356] "Generic (PLEG): container finished" podID="f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" containerID="3fdb13241b62ba27148dd495a23fec7938a6153cd285b9395104cd9ce178fe5c" exitCode=0
Nov 01 20:39:15.989335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:15.988831    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4" event={"ID":"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9","Type":"ContainerDied","Data":"3fdb13241b62ba27148dd495a23fec7938a6153cd285b9395104cd9ce178fe5c"}
Nov 01 20:39:17.421193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.421167    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:17.436181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.436145    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\") pod \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") "
Nov 01 20:39:17.436410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.436236    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\") pod \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") "
Nov 01 20:39:17.436410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.436283    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\" (UniqueName: \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\") pod \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\" (UID: \"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9\") "
Nov 01 20:39:17.436570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.436546    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca" pod "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" (UID: "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:17.439736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.439710    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968" pod "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" (UID: "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9"). InnerVolumeSpecName "kube-api-access-vr968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:17.440123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.440070    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca" pod "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" (UID: "f5af236d-1015-4e9e-bb8b-8ecb3366e2f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:17.537840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.537796    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:17.537840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.537830    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:17.537840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:17.537841    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr968\" (UniqueName: \"kubernetes.io/projected/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9-kube-api-access-vr968\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:18.021185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:18.021154    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"
Nov 01 20:39:18.024431 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:18.021314    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4" event={"ID":"f5af236d-1015-4e9e-bb8b-8ecb3366e2f9","Type":"ContainerDied","Data":"4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04"}
Nov 01 20:39:18.024820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:18.024790    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4197522672a5970dd3a37f9fa261d09853e4e1cd58a5a500f12aa9e1df4e0c04"
Nov 01 20:39:24.318280 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.318243    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"]
Nov 01 20:39:24.318710 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.318696    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" containerName="revoke-keys"
Nov 01 20:39:24.318774 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.318711    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" containerName="revoke-keys"
Nov 01 20:39:24.318878 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.318865    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" containerName="revoke-keys"
Nov 01 20:39:24.322276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.322256    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.344557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.344530    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:24.347514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.347488    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"]
Nov 01 20:39:24.355853 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.355828    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:24.355985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.355866    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:24.355985 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.355923    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:24.412928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.412895    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.413104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.412946    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtqj\" (UniqueName: \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.413104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.413035    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.513866 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.513830    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.514113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.513885    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtqj\" (UniqueName: \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.514113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.513912    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.514498 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.514476    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.516267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.516245    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.521463 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.521442    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtqj\" (UniqueName: \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\") pod \"maas-api-revoke-sub-78e063dd-2qnz9\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.635826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.635726    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:24.910136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:24.910109    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"]
Nov 01 20:39:24.911264 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:24.911233    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41925c63_45cb_49e4_9e14_90b5f3802c2e.slice/crio-e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc WatchSource:0}: Error finding container e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc: Status 404 returned error can't find the container with id e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc
Nov 01 20:39:25.061861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:25.061821    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9" event={"ID":"41925c63-45cb-49e4-9e14-90b5f3802c2e","Type":"ContainerStarted","Data":"e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc"}
Nov 01 20:39:26.074406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:26.074369    5041 generic.go:356] "Generic (PLEG): container finished" podID="41925c63-45cb-49e4-9e14-90b5f3802c2e" containerID="98f1db3bb38ebbbb61f82398c9b652e6a5a68b8571de055309c431d19c32f795" exitCode=0
Nov 01 20:39:26.075080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:26.074477    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9" event={"ID":"41925c63-45cb-49e4-9e14-90b5f3802c2e","Type":"ContainerDied","Data":"98f1db3bb38ebbbb61f82398c9b652e6a5a68b8571de055309c431d19c32f795"}
Nov 01 20:39:27.565068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.565038    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:27.669803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.669560    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\") pod \"41925c63-45cb-49e4-9e14-90b5f3802c2e\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") "
Nov 01 20:39:27.699802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.674010    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca" pod "41925c63-45cb-49e4-9e14-90b5f3802c2e" (UID: "41925c63-45cb-49e4-9e14-90b5f3802c2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:27.709800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.700464    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\" (UniqueName: \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\") pod \"41925c63-45cb-49e4-9e14-90b5f3802c2e\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") "
Nov 01 20:39:27.709800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.700600    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\") pod \"41925c63-45cb-49e4-9e14-90b5f3802c2e\" (UID: \"41925c63-45cb-49e4-9e14-90b5f3802c2e\") "
Nov 01 20:39:27.709800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.701012    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:27.726027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.725980    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj" pod "41925c63-45cb-49e4-9e14-90b5f3802c2e" (UID: "41925c63-45cb-49e4-9e14-90b5f3802c2e"). InnerVolumeSpecName "kube-api-access-2dtqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:27.727388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.727361    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca" pod "41925c63-45cb-49e4-9e14-90b5f3802c2e" (UID: "41925c63-45cb-49e4-9e14-90b5f3802c2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:27.807811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.805504    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41925c63-45cb-49e4-9e14-90b5f3802c2e-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:27.807811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:27.805541    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtqj\" (UniqueName: \"kubernetes.io/projected/41925c63-45cb-49e4-9e14-90b5f3802c2e-kube-api-access-2dtqj\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:28.127903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:28.127823    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"
Nov 01 20:39:28.128080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:28.127814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9" event={"ID":"41925c63-45cb-49e4-9e14-90b5f3802c2e","Type":"ContainerDied","Data":"e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc"}
Nov 01 20:39:28.128080 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:28.127956    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c5c332329becb436232fc2fafc74c20f9d4033aa6b7618c001cea546d4c0cc"
Nov 01 20:39:28.544053 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:39:28.544011    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:39:34.492083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.492048    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"]
Nov 01 20:39:34.492527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.492512    5041 cpu_manager.go:414] "containerMap: removing container" podUID="41925c63-45cb-49e4-9e14-90b5f3802c2e" containerName="revoke-keys"
Nov 01 20:39:34.492573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.492527    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41925c63-45cb-49e4-9e14-90b5f3802c2e" containerName="revoke-keys"
Nov 01 20:39:34.492639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.492630    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="41925c63-45cb-49e4-9e14-90b5f3802c2e" containerName="revoke-keys"
Nov 01 20:39:34.495852 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.495835    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.503654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.502903    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:34.503654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.503031    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:34.503654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.503135    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:34.503654 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.503182    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:34.505743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.505719    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"]
Nov 01 20:39:34.584069 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.584042    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp54h\" (UniqueName: \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.584161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.584140    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.584231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.584219    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.684995 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.684959    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp54h\" (UniqueName: \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.685177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.685039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.685177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.685083    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.685805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.685752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.687638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.687613    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.692858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.692835    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp54h\" (UniqueName: \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\") pod \"maas-api-revoke-sub-4789b905-v6mg8\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:34.809907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:34.809873    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:35.085631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:35.085605    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"]
Nov 01 20:39:35.088049 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:35.088019    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c6aa8_f76f_4728_be81_defa54c81326.slice/crio-3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777 WatchSource:0}: Error finding container 3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777: Status 404 returned error can't find the container with id 3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777
Nov 01 20:39:35.173643 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:35.173603    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8" event={"ID":"160c6aa8-f76f-4728-be81-defa54c81326","Type":"ContainerStarted","Data":"3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777"}
Nov 01 20:39:36.184417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:36.184376    5041 generic.go:356] "Generic (PLEG): container finished" podID="160c6aa8-f76f-4728-be81-defa54c81326" containerID="2cf3166e52faa3ac8b8c2d0cf71f3704b45498bf2d5d09ff481686b52974160e" exitCode=0
Nov 01 20:39:36.184936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:36.184454    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8" event={"ID":"160c6aa8-f76f-4728-be81-defa54c81326","Type":"ContainerDied","Data":"2cf3166e52faa3ac8b8c2d0cf71f3704b45498bf2d5d09ff481686b52974160e"}
Nov 01 20:39:37.125278 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.125241    5041 scope.go:122] "RemoveContainer" containerID="dac5080c2877c4fc4913dfc51a84a065b5457a2bb159fb650491f20ce5e260b3"
Nov 01 20:39:37.137297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.137273    5041 scope.go:122] "RemoveContainer" containerID="51328d292ba8ebf3e97bdef7e8bf6adfaeedb285d5c072d86ad12e42fa841092"
Nov 01 20:39:37.698015 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.697931    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:37.733708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.733669    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\") pod \"160c6aa8-f76f-4728-be81-defa54c81326\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") "
Nov 01 20:39:37.733911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.733745    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\" (UniqueName: \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\") pod \"160c6aa8-f76f-4728-be81-defa54c81326\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") "
Nov 01 20:39:37.733911 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.733802    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\") pod \"160c6aa8-f76f-4728-be81-defa54c81326\" (UID: \"160c6aa8-f76f-4728-be81-defa54c81326\") "
Nov 01 20:39:37.734342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.734307    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca" pod "160c6aa8-f76f-4728-be81-defa54c81326" (UID: "160c6aa8-f76f-4728-be81-defa54c81326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:37.736370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.736337    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h" pod "160c6aa8-f76f-4728-be81-defa54c81326" (UID: "160c6aa8-f76f-4728-be81-defa54c81326"). InnerVolumeSpecName "kube-api-access-dp54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:37.736503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.736400    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca" pod "160c6aa8-f76f-4728-be81-defa54c81326" (UID: "160c6aa8-f76f-4728-be81-defa54c81326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:37.834833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.834768    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:37.834833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.834830    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp54h\" (UniqueName: \"kubernetes.io/projected/160c6aa8-f76f-4728-be81-defa54c81326-kube-api-access-dp54h\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:37.835042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:37.834847    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/160c6aa8-f76f-4728-be81-defa54c81326-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:38.230629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:38.230592    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8" event={"ID":"160c6aa8-f76f-4728-be81-defa54c81326","Type":"ContainerDied","Data":"3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777"}
Nov 01 20:39:38.230861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:38.230677    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f468ad052b1855626de42deca564bd09c44bad781e4dffe6d29c60ebe8b4777"
Nov 01 20:39:38.230861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:38.230732    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"
Nov 01 20:39:44.698117 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.698028    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"]
Nov 01 20:39:44.698752 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.698731    5041 cpu_manager.go:414] "containerMap: removing container" podUID="160c6aa8-f76f-4728-be81-defa54c81326" containerName="revoke-keys"
Nov 01 20:39:44.698885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.698755    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160c6aa8-f76f-4728-be81-defa54c81326" containerName="revoke-keys"
Nov 01 20:39:44.698977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.698963    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="160c6aa8-f76f-4728-be81-defa54c81326" containerName="revoke-keys"
Nov 01 20:39:44.702507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.702486    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.706865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.706825    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:44.707011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.706896    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:44.707011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.706986    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:44.707377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.707353    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:44.709629 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.709344    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"]
Nov 01 20:39:44.807390 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.807354    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.807573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.807396    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrqw\" (UniqueName: \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.807573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.807520    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.909115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.909083    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.909115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.909120    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrqw\" (UniqueName: \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.909342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.909164    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.909915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.909889    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.911569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.911550    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:44.916236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:44.916215    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrqw\" (UniqueName: \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\") pod \"maas-api-revoke-sub-01a87047-prfhj\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:45.017909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:45.017830    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:45.302329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:45.302301    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"]
Nov 01 20:39:45.303662 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:45.303634    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6fa16c_a993_481c_a898_eee2c75cc673.slice/crio-40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2 WatchSource:0}: Error finding container 40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2: Status 404 returned error can't find the container with id 40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2
Nov 01 20:39:46.281769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:46.281717    5041 generic.go:356] "Generic (PLEG): container finished" podID="8a6fa16c-a993-481c-a898-eee2c75cc673" containerID="913fd5f71caf7b2201269472e5f2d8d78af3aff61483ea634b8397b4f330cd36" exitCode=0
Nov 01 20:39:46.281769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:46.281795    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj" event={"ID":"8a6fa16c-a993-481c-a898-eee2c75cc673","Type":"ContainerDied","Data":"913fd5f71caf7b2201269472e5f2d8d78af3aff61483ea634b8397b4f330cd36"}
Nov 01 20:39:46.281769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:46.281848    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj" event={"ID":"8a6fa16c-a993-481c-a898-eee2c75cc673","Type":"ContainerStarted","Data":"40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2"}
Nov 01 20:39:47.728113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.727997    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:47.738845 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.738819    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\" (UniqueName: \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\") pod \"8a6fa16c-a993-481c-a898-eee2c75cc673\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") "
Nov 01 20:39:47.739032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.738949    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\") pod \"8a6fa16c-a993-481c-a898-eee2c75cc673\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") "
Nov 01 20:39:47.739032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.739021    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\") pod \"8a6fa16c-a993-481c-a898-eee2c75cc673\" (UID: \"8a6fa16c-a993-481c-a898-eee2c75cc673\") "
Nov 01 20:39:47.739384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.739346    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca" pod "8a6fa16c-a993-481c-a898-eee2c75cc673" (UID: "8a6fa16c-a993-481c-a898-eee2c75cc673"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:47.739526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.739416    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:47.741061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.741032    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca" pod "8a6fa16c-a993-481c-a898-eee2c75cc673" (UID: "8a6fa16c-a993-481c-a898-eee2c75cc673"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:47.741188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.741093    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw" pod "8a6fa16c-a993-481c-a898-eee2c75cc673" (UID: "8a6fa16c-a993-481c-a898-eee2c75cc673"). InnerVolumeSpecName "kube-api-access-vhrqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:47.841808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.841004    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6fa16c-a993-481c-a898-eee2c75cc673-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:47.841808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:47.841042    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrqw\" (UniqueName: \"kubernetes.io/projected/8a6fa16c-a993-481c-a898-eee2c75cc673-kube-api-access-vhrqw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:48.320807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:48.319587    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj" event={"ID":"8a6fa16c-a993-481c-a898-eee2c75cc673","Type":"ContainerDied","Data":"40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2"}
Nov 01 20:39:48.320807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:48.319653    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40834c0fde706ce9d19bb813eefdff3221a4be0fc560dca2ab9289d17ce3fcc2"
Nov 01 20:39:48.320807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:48.319724    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"
Nov 01 20:39:48.353800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:48.351871    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:39:48.353800 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:48.351933    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:39:55.001118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.001079    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"]
Nov 01 20:39:55.001567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.001554    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8a6fa16c-a993-481c-a898-eee2c75cc673" containerName="revoke-keys"
Nov 01 20:39:55.001618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.001568    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6fa16c-a993-481c-a898-eee2c75cc673" containerName="revoke-keys"
Nov 01 20:39:55.001688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.001679    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6fa16c-a993-481c-a898-eee2c75cc673" containerName="revoke-keys"
Nov 01 20:39:55.005147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.005128    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.009340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.009090    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:39:55.009340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.009107    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:39:55.009340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.009186    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:39:55.009757 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.009734    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:39:55.012046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.012024    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"]
Nov 01 20:39:55.161522 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.161488    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.161708 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.161648    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.161816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.161710    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8jl\" (UniqueName: \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.263001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.262909    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.263190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.263020    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.263190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.263050    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8jl\" (UniqueName: \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.263914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.263886    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.265402 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.265372    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.270925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.270894    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8jl\" (UniqueName: \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\") pod \"maas-api-revoke-sub-4158a440-w84tt\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.322546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.322516    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:55.598922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:55.598889    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"]
Nov 01 20:39:55.600114 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:39:55.600089    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76201f1e_718a_49d2_ac98_92d109b0961f.slice/crio-43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83 WatchSource:0}: Error finding container 43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83: Status 404 returned error can't find the container with id 43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83
Nov 01 20:39:56.374275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:56.374217    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt" event={"ID":"76201f1e-718a-49d2-ac98-92d109b0961f","Type":"ContainerStarted","Data":"8a4853851b15a0bea0083a64339495a83862936b5cb1727a860ac94db8967e1a"}
Nov 01 20:39:56.374882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:56.374298    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt" event={"ID":"76201f1e-718a-49d2-ac98-92d109b0961f","Type":"ContainerStarted","Data":"43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83"}
Nov 01 20:39:56.391618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:56.391554    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt" podStartSLOduration=2.391535766 podStartE2EDuration="2.391535766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:39:56.390148901 +0000 UTC m=+2484.111227578" watchObservedRunningTime="2026-11-01 20:39:56.391535766 +0000 UTC m=+2484.112614443"
Nov 01 20:39:57.416621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:57.415462    5041 generic.go:356] "Generic (PLEG): container finished" podID="76201f1e-718a-49d2-ac98-92d109b0961f" containerID="8a4853851b15a0bea0083a64339495a83862936b5cb1727a860ac94db8967e1a" exitCode=0
Nov 01 20:39:57.416621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:57.415571    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt" event={"ID":"76201f1e-718a-49d2-ac98-92d109b0961f","Type":"ContainerDied","Data":"8a4853851b15a0bea0083a64339495a83862936b5cb1727a860ac94db8967e1a"}
Nov 01 20:39:58.639505 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:39:58.639399    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:39:58.822729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.822701    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.829298    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\") pod \"76201f1e-718a-49d2-ac98-92d109b0961f\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") "
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.829743    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\") pod \"76201f1e-718a-49d2-ac98-92d109b0961f\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") "
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.830103    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca" pod "76201f1e-718a-49d2-ac98-92d109b0961f" (UID: "76201f1e-718a-49d2-ac98-92d109b0961f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.830273    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\" (UniqueName: \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\") pod \"76201f1e-718a-49d2-ac98-92d109b0961f\" (UID: \"76201f1e-718a-49d2-ac98-92d109b0961f\") "
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.830986    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.832152    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca" pod "76201f1e-718a-49d2-ac98-92d109b0961f" (UID: "76201f1e-718a-49d2-ac98-92d109b0961f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:39:58.836809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.834025    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl" pod "76201f1e-718a-49d2-ac98-92d109b0961f" (UID: "76201f1e-718a-49d2-ac98-92d109b0961f"). InnerVolumeSpecName "kube-api-access-8r8jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:39:58.932799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.932072    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8jl\" (UniqueName: \"kubernetes.io/projected/76201f1e-718a-49d2-ac98-92d109b0961f-kube-api-access-8r8jl\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:58.932799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:58.932103    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76201f1e-718a-49d2-ac98-92d109b0961f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:39:59.438221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:59.438175    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt" event={"ID":"76201f1e-718a-49d2-ac98-92d109b0961f","Type":"ContainerDied","Data":"43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83"}
Nov 01 20:39:59.438557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:59.438534    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ef49f95c7f607f95cde17cd37e5f7039b1191ba42bf0faac7ab2896b3efc83"
Nov 01 20:39:59.438760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:39:59.438747    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"
Nov 01 20:40:05.206410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.206368    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"]
Nov 01 20:40:05.206943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.206925    5041 cpu_manager.go:414] "containerMap: removing container" podUID="76201f1e-718a-49d2-ac98-92d109b0961f" containerName="revoke-keys"
Nov 01 20:40:05.206943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.206943    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76201f1e-718a-49d2-ac98-92d109b0961f" containerName="revoke-keys"
Nov 01 20:40:05.207038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.207021    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="76201f1e-718a-49d2-ac98-92d109b0961f" containerName="revoke-keys"
Nov 01 20:40:05.210392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.210376    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.214394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.214363    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:40:05.214526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.214404    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:40:05.214526 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.214506    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:40:05.215276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.215249    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:40:05.216814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.216790    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"]
Nov 01 20:40:05.299980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.299947    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.300159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.299993    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn679\" (UniqueName: \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.300159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.300078    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.401560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.401519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.401745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.401568    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn679\" (UniqueName: \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.401745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.401601    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.402302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.402281    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.403994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.403971    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.410393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.410370    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn679\" (UniqueName: \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\") pod \"maas-api-revoke-sub-ff92358e-9lpvm\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.527106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.527023    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:05.804345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:05.804316    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"]
Nov 01 20:40:05.805096 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:40:05.805067    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114f165d_d59f_4c88_9581_af2e8d2aa53f.slice/crio-01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf WatchSource:0}: Error finding container 01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf: Status 404 returned error can't find the container with id 01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf
Nov 01 20:40:06.484398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:06.484309    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm" event={"ID":"114f165d-d59f-4c88-9581-af2e8d2aa53f","Type":"ContainerStarted","Data":"ebf9aeaaea5e655fd2b5f0eeccb3e0dce72823a52858d700416d4d56544046ff"}
Nov 01 20:40:06.484398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:06.484363    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm" event={"ID":"114f165d-d59f-4c88-9581-af2e8d2aa53f","Type":"ContainerStarted","Data":"01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf"}
Nov 01 20:40:06.500840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:06.500770    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm" podStartSLOduration=1.500756317 podStartE2EDuration="1.500756317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:06.498626128 +0000 UTC m=+2494.219704806" watchObservedRunningTime="2026-11-01 20:40:06.500756317 +0000 UTC m=+2494.221835046"
Nov 01 20:40:07.508534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:07.508497    5041 generic.go:356] "Generic (PLEG): container finished" podID="114f165d-d59f-4c88-9581-af2e8d2aa53f" containerID="ebf9aeaaea5e655fd2b5f0eeccb3e0dce72823a52858d700416d4d56544046ff" exitCode=0
Nov 01 20:40:07.509149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:07.509123    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm" event={"ID":"114f165d-d59f-4c88-9581-af2e8d2aa53f","Type":"ContainerDied","Data":"ebf9aeaaea5e655fd2b5f0eeccb3e0dce72823a52858d700416d4d56544046ff"}
Nov 01 20:40:09.034540 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.034506    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:09.171928 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.171891    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\") pod \"114f165d-d59f-4c88-9581-af2e8d2aa53f\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") "
Nov 01 20:40:09.172120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.172026    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\") pod \"114f165d-d59f-4c88-9581-af2e8d2aa53f\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") "
Nov 01 20:40:09.172120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.172090    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\" (UniqueName: \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\") pod \"114f165d-d59f-4c88-9581-af2e8d2aa53f\" (UID: \"114f165d-d59f-4c88-9581-af2e8d2aa53f\") "
Nov 01 20:40:09.172538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.172506    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca" pod "114f165d-d59f-4c88-9581-af2e8d2aa53f" (UID: "114f165d-d59f-4c88-9581-af2e8d2aa53f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:09.174740 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.174708    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679" pod "114f165d-d59f-4c88-9581-af2e8d2aa53f" (UID: "114f165d-d59f-4c88-9581-af2e8d2aa53f"). InnerVolumeSpecName "kube-api-access-pn679". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:09.174899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.174863    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca" pod "114f165d-d59f-4c88-9581-af2e8d2aa53f" (UID: "114f165d-d59f-4c88-9581-af2e8d2aa53f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:09.273272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.273190    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:09.273272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.273219    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114f165d-d59f-4c88-9581-af2e8d2aa53f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:09.273272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.273228    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn679\" (UniqueName: \"kubernetes.io/projected/114f165d-d59f-4c88-9581-af2e8d2aa53f-kube-api-access-pn679\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:09.527831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.527725    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"
Nov 01 20:40:09.527831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.527724    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm" event={"ID":"114f165d-d59f-4c88-9581-af2e8d2aa53f","Type":"ContainerDied","Data":"01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf"}
Nov 01 20:40:09.528027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:09.527851    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01183462ed427eaf285024b7b77301a2f53c6673666724c5f1afe9bb111fa1bf"
Nov 01 20:40:15.411665 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.411623    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"]
Nov 01 20:40:15.412349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.412325    5041 cpu_manager.go:414] "containerMap: removing container" podUID="114f165d-d59f-4c88-9581-af2e8d2aa53f" containerName="revoke-keys"
Nov 01 20:40:15.412461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.412352    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114f165d-d59f-4c88-9581-af2e8d2aa53f" containerName="revoke-keys"
Nov 01 20:40:15.412516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.412472    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="114f165d-d59f-4c88-9581-af2e8d2aa53f" containerName="revoke-keys"
Nov 01 20:40:15.415963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.415940    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.422881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.420965    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:40:15.422881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.421500    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:40:15.422881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.421647    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:40:15.422881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.421721    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:40:15.425470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.425437    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"]
Nov 01 20:40:15.536691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.536654    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgrg\" (UniqueName: \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.536902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.536860    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.537010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.536988    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.638257 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.638214    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgrg\" (UniqueName: \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.638519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.638311    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.638519 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.638340    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.639055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.639024    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.640725 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.640702    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.646906 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.646883    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgrg\" (UniqueName: \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\") pod \"maas-api-revoke-sub-89db3ea8-wtm9r\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:15.743150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:15.743061    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:16.023618 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:16.023587    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"]
Nov 01 20:40:16.024472 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:40:16.024442    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf128807e_d623_4986_9e4f_def79e80a71a.slice/crio-b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa WatchSource:0}: Error finding container b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa: Status 404 returned error can't find the container with id b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa
Nov 01 20:40:16.580639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:16.580591    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r" event={"ID":"f128807e-d623-4986-9e4f-def79e80a71a","Type":"ContainerStarted","Data":"741bb4d343c309d4f5b32478c8091ed0ab7845a5af4e7ce6b4ae00b0e53d9f46"}
Nov 01 20:40:16.581076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:16.580666    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r" event={"ID":"f128807e-d623-4986-9e4f-def79e80a71a","Type":"ContainerStarted","Data":"b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa"}
Nov 01 20:40:16.600497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:16.600445    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r" podStartSLOduration=1.600431223 podStartE2EDuration="1.600431223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:16.597206828 +0000 UTC m=+2504.318285503" watchObservedRunningTime="2026-11-01 20:40:16.600431223 +0000 UTC m=+2504.321509935"
Nov 01 20:40:17.606808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:17.603845    5041 generic.go:356] "Generic (PLEG): container finished" podID="f128807e-d623-4986-9e4f-def79e80a71a" containerID="741bb4d343c309d4f5b32478c8091ed0ab7845a5af4e7ce6b4ae00b0e53d9f46" exitCode=0
Nov 01 20:40:17.606808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:17.604049    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r" event={"ID":"f128807e-d623-4986-9e4f-def79e80a71a","Type":"ContainerDied","Data":"741bb4d343c309d4f5b32478c8091ed0ab7845a5af4e7ce6b4ae00b0e53d9f46"}
Nov 01 20:40:18.340916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.340880    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-bnsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 20:40:18.341105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.340935    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-bnsdt" podUID="b1388a62-2172-466e-91f6-dcef2cf7525b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 20:40:18.790370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.790332    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:18.832256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.832224    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\" (UniqueName: \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\") pod \"f128807e-d623-4986-9e4f-def79e80a71a\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") "
Nov 01 20:40:18.832465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.832270    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\") pod \"f128807e-d623-4986-9e4f-def79e80a71a\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") "
Nov 01 20:40:18.832465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.832333    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\") pod \"f128807e-d623-4986-9e4f-def79e80a71a\" (UID: \"f128807e-d623-4986-9e4f-def79e80a71a\") "
Nov 01 20:40:18.832711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.832685    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca" pod "f128807e-d623-4986-9e4f-def79e80a71a" (UID: "f128807e-d623-4986-9e4f-def79e80a71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:18.832849 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.832828    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:18.834444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.834418    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca" pod "f128807e-d623-4986-9e4f-def79e80a71a" (UID: "f128807e-d623-4986-9e4f-def79e80a71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:18.834973 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.834952    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg" pod "f128807e-d623-4986-9e4f-def79e80a71a" (UID: "f128807e-d623-4986-9e4f-def79e80a71a"). InnerVolumeSpecName "kube-api-access-6bgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:18.934375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.934290    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bgrg\" (UniqueName: \"kubernetes.io/projected/f128807e-d623-4986-9e4f-def79e80a71a-kube-api-access-6bgrg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:18.934375 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:18.934326    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f128807e-d623-4986-9e4f-def79e80a71a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:19.621582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:19.621552    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"
Nov 01 20:40:19.621825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:19.621587    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r" event={"ID":"f128807e-d623-4986-9e4f-def79e80a71a","Type":"ContainerDied","Data":"b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa"}
Nov 01 20:40:19.621825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:19.621638    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4508781867825f1dc035efcce552a98cf26177a608e38a2fd86cb6881bdfdfa"
Nov 01 20:40:28.063407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.063366    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"]
Nov 01 20:40:28.081360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.081325    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f128807e-d623-4986-9e4f-def79e80a71a" containerName="revoke-keys"
Nov 01 20:40:28.081610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.081589    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f128807e-d623-4986-9e4f-def79e80a71a" containerName="revoke-keys"
Nov 01 20:40:28.081945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.081927    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f128807e-d623-4986-9e4f-def79e80a71a" containerName="revoke-keys"
Nov 01 20:40:28.122060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.122027    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"]
Nov 01 20:40:28.122409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.122384    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.129093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.128864    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:40:28.129093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.128964    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:40:28.129093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.129042    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:40:28.129093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.129066    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:40:28.132187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.131887    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.132187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.131975    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xw5z\" (UniqueName: \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.132187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.132066    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.233118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.233076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.233324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.233135    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.233324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.233189    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xw5z\" (UniqueName: \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.233754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.233725    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.237329 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.237307    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.241374 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.241349    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xw5z\" (UniqueName: \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\") pod \"maas-api-revoke-sub-69d126ad-v9smk\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.527255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.527159    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:28.626815 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:40:28.620994    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:40:28.864819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:28.864092    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"]
Nov 01 20:40:29.709348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:29.709304    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk" event={"ID":"20a8718e-7be0-44d4-a4bd-e0113cca2201","Type":"ContainerStarted","Data":"3dcccc391f47ccb3190d5074a312081f48290fb453685c551f1309c7fceb93f3"}
Nov 01 20:40:29.709766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:29.709371    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk" event={"ID":"20a8718e-7be0-44d4-a4bd-e0113cca2201","Type":"ContainerStarted","Data":"4b6d2169002e0c55c97f7df51fdf53bf5cb24298d87bf5728b668c56175a98a9"}
Nov 01 20:40:29.728358 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:29.728309    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk" podStartSLOduration=1.728295536 podStartE2EDuration="1.728295536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:40:29.725397713 +0000 UTC m=+2517.446476388" watchObservedRunningTime="2026-11-01 20:40:29.728295536 +0000 UTC m=+2517.449374211"
Nov 01 20:40:30.717554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:30.717517    5041 generic.go:356] "Generic (PLEG): container finished" podID="20a8718e-7be0-44d4-a4bd-e0113cca2201" containerID="3dcccc391f47ccb3190d5074a312081f48290fb453685c551f1309c7fceb93f3" exitCode=0
Nov 01 20:40:30.718009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:30.717595    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk" event={"ID":"20a8718e-7be0-44d4-a4bd-e0113cca2201","Type":"ContainerDied","Data":"3dcccc391f47ccb3190d5074a312081f48290fb453685c551f1309c7fceb93f3"}
Nov 01 20:40:31.873118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.873093    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:31.964693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.964657    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\" (UniqueName: \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\") pod \"20a8718e-7be0-44d4-a4bd-e0113cca2201\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") "
Nov 01 20:40:31.964914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.964755    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\") pod \"20a8718e-7be0-44d4-a4bd-e0113cca2201\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") "
Nov 01 20:40:31.964914 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.964802    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\") pod \"20a8718e-7be0-44d4-a4bd-e0113cca2201\" (UID: \"20a8718e-7be0-44d4-a4bd-e0113cca2201\") "
Nov 01 20:40:31.965244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.965209    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca" pod "20a8718e-7be0-44d4-a4bd-e0113cca2201" (UID: "20a8718e-7be0-44d4-a4bd-e0113cca2201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:31.966962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.966926    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca" pod "20a8718e-7be0-44d4-a4bd-e0113cca2201" (UID: "20a8718e-7be0-44d4-a4bd-e0113cca2201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:31.967093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:31.967043    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z" pod "20a8718e-7be0-44d4-a4bd-e0113cca2201" (UID: "20a8718e-7be0-44d4-a4bd-e0113cca2201"). InnerVolumeSpecName "kube-api-access-4xw5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:32.065598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.065561    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:32.065598 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.065594    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a8718e-7be0-44d4-a4bd-e0113cca2201-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:32.065850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.065607    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xw5z\" (UniqueName: \"kubernetes.io/projected/20a8718e-7be0-44d4-a4bd-e0113cca2201-kube-api-access-4xw5z\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:32.735466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.735391    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"
Nov 01 20:40:32.735466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.735405    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk" event={"ID":"20a8718e-7be0-44d4-a4bd-e0113cca2201","Type":"ContainerDied","Data":"4b6d2169002e0c55c97f7df51fdf53bf5cb24298d87bf5728b668c56175a98a9"}
Nov 01 20:40:32.735466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:32.735458    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6d2169002e0c55c97f7df51fdf53bf5cb24298d87bf5728b668c56175a98a9"
Nov 01 20:40:41.026028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:41.025986    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"]
Nov 01 20:40:41.029131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:41.029088    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-vsjxd"]
Nov 01 20:40:42.427631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:42.427592    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d54bea-2476-4c0b-a6c6-bd3812fb6998" path="/var/lib/kubelet/pods/f6d54bea-2476-4c0b-a6c6-bd3812fb6998/volumes"
Nov 01 20:40:48.087808 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.087759    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"]
Nov 01 20:40:48.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.088253    5041 cpu_manager.go:414] "containerMap: removing container" podUID="20a8718e-7be0-44d4-a4bd-e0113cca2201" containerName="revoke-keys"
Nov 01 20:40:48.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.088264    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a8718e-7be0-44d4-a4bd-e0113cca2201" containerName="revoke-keys"
Nov 01 20:40:48.090241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.088365    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a8718e-7be0-44d4-a4bd-e0113cca2201" containerName="revoke-keys"
Nov 01 20:40:48.091523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.091507    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.096234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.096198    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:40:48.096234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.096213    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:40:48.096461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.096263    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:40:48.096461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.096272    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:40:48.100496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.100004    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"]
Nov 01 20:40:48.230221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.230181    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjvc\" (UniqueName: \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.230413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.230240    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.230413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.230350    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.331095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.331047    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjvc\" (UniqueName: \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.331312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.331121    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.331312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.331181    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.332006 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.331981    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.333831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.333806    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.338483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.338425    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjvc\" (UniqueName: \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\") pod \"maas-api-revoke-sub-683fc09a-9jdsf\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.409131 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.409098    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:48.692745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.692720    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"]
Nov 01 20:40:48.694987 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:40:48.694959    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059f4463_d71a_437b_b495_7b7baba3b061.slice/crio-4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0 WatchSource:0}: Error finding container 4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0: Status 404 returned error can't find the container with id 4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0
Nov 01 20:40:48.833471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:48.833432    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf" event={"ID":"059f4463-d71a-437b-b495-7b7baba3b061","Type":"ContainerStarted","Data":"4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0"}
Nov 01 20:40:49.841918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:49.841817    5041 generic.go:356] "Generic (PLEG): container finished" podID="059f4463-d71a-437b-b495-7b7baba3b061" containerID="53196fa755567c9a569cb455c3f75f39ee64a768db0582d9a2f52b649020ba23" exitCode=0
Nov 01 20:40:49.841918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:49.841900    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf" event={"ID":"059f4463-d71a-437b-b495-7b7baba3b061","Type":"ContainerDied","Data":"53196fa755567c9a569cb455c3f75f39ee64a768db0582d9a2f52b649020ba23"}
Nov 01 20:40:51.013580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.013552    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:51.161138 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.161054    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\") pod \"059f4463-d71a-437b-b495-7b7baba3b061\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") "
Nov 01 20:40:51.161288 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.161176    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\" (UniqueName: \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\") pod \"059f4463-d71a-437b-b495-7b7baba3b061\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") "
Nov 01 20:40:51.161387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.161285    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\") pod \"059f4463-d71a-437b-b495-7b7baba3b061\" (UID: \"059f4463-d71a-437b-b495-7b7baba3b061\") "
Nov 01 20:40:51.161688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.161653    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca" pod "059f4463-d71a-437b-b495-7b7baba3b061" (UID: "059f4463-d71a-437b-b495-7b7baba3b061"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:40:51.163261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.163231    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca" pod "059f4463-d71a-437b-b495-7b7baba3b061" (UID: "059f4463-d71a-437b-b495-7b7baba3b061"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:40:51.163359 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.163298    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc" pod "059f4463-d71a-437b-b495-7b7baba3b061" (UID: "059f4463-d71a-437b-b495-7b7baba3b061"). InnerVolumeSpecName "kube-api-access-xpjvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:40:51.263029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.262985    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:51.263029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.263017    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpjvc\" (UniqueName: \"kubernetes.io/projected/059f4463-d71a-437b-b495-7b7baba3b061-kube-api-access-xpjvc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:51.263029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.263028    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f4463-d71a-437b-b495-7b7baba3b061-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:40:51.856997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.856965    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"
Nov 01 20:40:51.857195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.856999    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf" event={"ID":"059f4463-d71a-437b-b495-7b7baba3b061","Type":"ContainerDied","Data":"4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0"}
Nov 01 20:40:51.857195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:40:51.857050    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4187170b77a98974c57a59fdd2566539cc6975e930a8f4d1080894abdda8c3c0"
Nov 01 20:40:58.540906 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:40:58.540869    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:41:11.419116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:11.419081    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:41:11.419655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:11.419314    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" containerID="cri-o://a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769" gracePeriod=30
Nov 01 20:41:15.210904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:15.210818    5041 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 20:41:15.210904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:15.210877    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 20:41:16.677345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.677318    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:41:16.730834 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.730742    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.730990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.730823    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.730990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.730898    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.730990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.730953    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731002    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731031    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731076    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.731175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731132    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731182    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731212    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\") pod \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\" (UID: \"83c06eb5-1248-46b7-8c8f-ebb28eac050e\") "
Nov 01 20:41:16.731395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731311    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.731612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731588    5041 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-credential-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.731688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731610    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:16.731688 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731617    5041 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.731823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731700    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.731895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.731830    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.733442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.733411    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:16.733573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.733498    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:41:16.733638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.733622    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "kube-api-access-2dwqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:16.733701 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.733682    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo" pod "83c06eb5-1248-46b7-8c8f-ebb28eac050e" (UID: "83c06eb5-1248-46b7-8c8f-ebb28eac050e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 20:41:16.832592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832556    5041 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-data\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832586    5041 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-token\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832592 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832596    5041 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-podinfo\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832606    5041 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-workload-socket\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832618    5041 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istio-envoy\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832628    5041 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c06eb5-1248-46b7-8c8f-ebb28eac050e-istiod-ca-cert\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:16.832882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:16.832636    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwqd\" (UniqueName: \"kubernetes.io/projected/83c06eb5-1248-46b7-8c8f-ebb28eac050e-kube-api-access-2dwqd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:17.017047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.016963    5041 generic.go:356] "Generic (PLEG): container finished" podID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerID="a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769" exitCode=0
Nov 01 20:41:17.017047 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.017011    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"
Nov 01 20:41:17.017236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.017044    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" event={"ID":"83c06eb5-1248-46b7-8c8f-ebb28eac050e","Type":"ContainerDied","Data":"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"}
Nov 01 20:41:17.017236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.017099    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b" event={"ID":"83c06eb5-1248-46b7-8c8f-ebb28eac050e","Type":"ContainerDied","Data":"b0cd7b4525bff40fd26d670e32da4eccae3123352c282feee6da6025b016a51b"}
Nov 01 20:41:17.017236 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.017125    5041 scope.go:122] "RemoveContainer" containerID="a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"
Nov 01 20:41:17.027768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.027752    5041 scope.go:122] "RemoveContainer" containerID="a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"
Nov 01 20:41:17.028022 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:41:17.028005    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769\": container with ID starting with a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769 not found: ID does not exist" containerID="a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"
Nov 01 20:41:17.028073 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.028031    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769"} err="failed to get container status \"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769\": rpc error: code = NotFound desc = could not find container \"a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769\": container with ID starting with a8b4c4902d0b31f56e119526f26c748254544137c44d7f009d0898790f31b769 not found: ID does not exist"
Nov 01 20:41:17.040083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.040058    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:41:17.044529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:17.044495    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-phl9b"]
Nov 01 20:41:18.436176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:18.436138    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" path="/var/lib/kubelet/pods/83c06eb5-1248-46b7-8c8f-ebb28eac050e/volumes"
Nov 01 20:41:28.541803 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:41:28.541749    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:41:33.982749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.982709    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"]
Nov 01 20:41:33.983213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983199    5041 cpu_manager.go:414] "containerMap: removing container" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy"
Nov 01 20:41:33.983259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983214    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy"
Nov 01 20:41:33.983259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983241    5041 cpu_manager.go:414] "containerMap: removing container" podUID="059f4463-d71a-437b-b495-7b7baba3b061" containerName="revoke-keys"
Nov 01 20:41:33.983259 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983246    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059f4463-d71a-437b-b495-7b7baba3b061" containerName="revoke-keys"
Nov 01 20:41:33.983356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983313    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="059f4463-d71a-437b-b495-7b7baba3b061" containerName="revoke-keys"
Nov 01 20:41:33.983356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.983326    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c06eb5-1248-46b7-8c8f-ebb28eac050e" containerName="istio-proxy"
Nov 01 20:41:33.986076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.986057    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:33.990434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.990411    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:41:33.991372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.991349    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\""
Nov 01 20:41:33.991538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.991391    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:41:33.991538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.991421    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:41:33.994811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:33.994762    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"]
Nov 01 20:41:34.133191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.133156    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.133368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.133296    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.133418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.133400    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.234056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.233962    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.234056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.234029    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.234287 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.234114    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.234674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.234654    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.236472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.236451    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.239259 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:41:34.239233    5041 projected.go:196] Error preparing data for projected volume kube-api-access-gkjsr for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 01 20:41:34.239351 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:41:34.239298    5041 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr podName:24392edb-b979-4ad9-8dc7-44d8b9e1cc48 nodeName:}" failed. No retries permitted until 2026-11-01 20:41:34.73928305 +0000 UTC m=+2582.460361717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gkjsr" (UniqueName: "kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr") pod "maas-api-revoke-keys-e2e-ait-022966d4-42zk7" (UID: "24392edb-b979-4ad9-8dc7-44d8b9e1cc48") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 01 20:41:34.839321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.839224    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.847848 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.847818    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") pod \"maas-api-revoke-keys-e2e-ait-022966d4-42zk7\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:34.900647 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.900616    5041 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7" secret="" err="secret \"maas-api-cleanup-dockercfg-tq9vg\" not found"
Nov 01 20:41:34.900864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:34.900666    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:35.179744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:35.179718    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"]
Nov 01 20:41:35.181969 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:41:35.181938    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24392edb_b979_4ad9_8dc7_44d8b9e1cc48.slice/crio-0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd WatchSource:0}: Error finding container 0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd: Status 404 returned error can't find the container with id 0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd
Nov 01 20:41:36.134836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:36.134802    5041 generic.go:356] "Generic (PLEG): container finished" podID="24392edb-b979-4ad9-8dc7-44d8b9e1cc48" containerID="a736ca29159166b25e8890080813333e31b515c9b667b91d2f71c9c0b3b69df2" exitCode=0
Nov 01 20:41:36.135024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:36.134877    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7" event={"ID":"24392edb-b979-4ad9-8dc7-44d8b9e1cc48","Type":"ContainerDied","Data":"a736ca29159166b25e8890080813333e31b515c9b667b91d2f71c9c0b3b69df2"}
Nov 01 20:41:36.135024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:36.134915    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7" event={"ID":"24392edb-b979-4ad9-8dc7-44d8b9e1cc48","Type":"ContainerStarted","Data":"0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd"}
Nov 01 20:41:37.286409 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.286382    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:37.467012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.466914    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") pod \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") "
Nov 01 20:41:37.467012 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.466969    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\") pod \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") "
Nov 01 20:41:37.467231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.467060    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\") pod \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\" (UID: \"24392edb-b979-4ad9-8dc7-44d8b9e1cc48\") "
Nov 01 20:41:37.467361 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.467339    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca" pod "24392edb-b979-4ad9-8dc7-44d8b9e1cc48" (UID: "24392edb-b979-4ad9-8dc7-44d8b9e1cc48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:41:37.469279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.469243    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca" pod "24392edb-b979-4ad9-8dc7-44d8b9e1cc48" (UID: "24392edb-b979-4ad9-8dc7-44d8b9e1cc48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:41:37.469410 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.469393    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr" pod "24392edb-b979-4ad9-8dc7-44d8b9e1cc48" (UID: "24392edb-b979-4ad9-8dc7-44d8b9e1cc48"). InnerVolumeSpecName "kube-api-access-gkjsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:41:37.568299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.568266    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:37.568299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.568298    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkjsr\" (UniqueName: \"kubernetes.io/projected/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-kube-api-access-gkjsr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:37.568299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:37.568313    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24392edb-b979-4ad9-8dc7-44d8b9e1cc48-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:41:38.149721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:38.149690    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"
Nov 01 20:41:38.149895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:38.149694    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7" event={"ID":"24392edb-b979-4ad9-8dc7-44d8b9e1cc48","Type":"ContainerDied","Data":"0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd"}
Nov 01 20:41:38.149895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:38.149841    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8f883a3272986bcd8b7ae47edc6ecdc594a25ec0d158d9988e53c91f3bfccd"
Nov 01 20:41:45.305349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:45.305305    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"]
Nov 01 20:41:45.308228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:45.308200    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-022966d4-42zk7"]
Nov 01 20:41:46.427686 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:46.427651    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24392edb-b979-4ad9-8dc7-44d8b9e1cc48" path="/var/lib/kubelet/pods/24392edb-b979-4ad9-8dc7-44d8b9e1cc48/volumes"
Nov 01 20:41:56.972197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.972163    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"]
Nov 01 20:41:56.972679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.972652    5041 cpu_manager.go:414] "containerMap: removing container" podUID="24392edb-b979-4ad9-8dc7-44d8b9e1cc48" containerName="revoke-keys"
Nov 01 20:41:56.972679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.972664    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24392edb-b979-4ad9-8dc7-44d8b9e1cc48" containerName="revoke-keys"
Nov 01 20:41:56.972754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.972750    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="24392edb-b979-4ad9-8dc7-44d8b9e1cc48" containerName="revoke-keys"
Nov 01 20:41:56.976373 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.976356    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:56.980198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.980176    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:41:56.980737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.980717    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:41:56.980897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.980833    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:41:56.980975 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.980913    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:41:56.983454 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:56.983428    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"]
Nov 01 20:41:57.098243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.098203    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprs6\" (UniqueName: \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.098439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.098255    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.098524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.098455    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.199294 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.199256    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.199514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.199324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprs6\" (UniqueName: \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.199514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.199355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.200027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.200004    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.201821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.201799    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.207026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.207003    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprs6\" (UniqueName: \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\") pod \"maas-api-revoke-sub-f904bff9-sfk74\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.293120 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.293086    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:41:57.567666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:57.567635    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"]
Nov 01 20:41:57.568659 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:41:57.568634    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2ffa87_b809_41d2_bea3_6e870f3e9969.slice/crio-40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a WatchSource:0}: Error finding container 40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a: Status 404 returned error can't find the container with id 40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a
Nov 01 20:41:58.289736 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:58.289701    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74" event={"ID":"6c2ffa87-b809-41d2-bea3-6e870f3e9969","Type":"ContainerStarted","Data":"c06b97fe8076085c1bfaa08f79c967980e73eb5a4baf73f9d634bf746f8320ea"}
Nov 01 20:41:58.290153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:58.289765    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74" event={"ID":"6c2ffa87-b809-41d2-bea3-6e870f3e9969","Type":"ContainerStarted","Data":"40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a"}
Nov 01 20:41:58.306694 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:58.306646    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74" podStartSLOduration=2.30663177 podStartE2EDuration="2.30663177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:58.304705622 +0000 UTC m=+2606.025784298" watchObservedRunningTime="2026-11-01 20:41:58.30663177 +0000 UTC m=+2606.027710446"
Nov 01 20:41:58.540814 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:41:58.540699    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:41:59.297712 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:59.297675    5041 generic.go:356] "Generic (PLEG): container finished" podID="6c2ffa87-b809-41d2-bea3-6e870f3e9969" containerID="c06b97fe8076085c1bfaa08f79c967980e73eb5a4baf73f9d634bf746f8320ea" exitCode=0
Nov 01 20:41:59.298170 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:41:59.297763    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74" event={"ID":"6c2ffa87-b809-41d2-bea3-6e870f3e9969","Type":"ContainerDied","Data":"c06b97fe8076085c1bfaa08f79c967980e73eb5a4baf73f9d634bf746f8320ea"}
Nov 01 20:42:00.450166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.450143    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:42:00.534102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.534067    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\") pod \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") "
Nov 01 20:42:00.534293 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.534255    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\" (UniqueName: \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\") pod \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") "
Nov 01 20:42:00.534365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.534299    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\") pod \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\" (UID: \"6c2ffa87-b809-41d2-bea3-6e870f3e9969\") "
Nov 01 20:42:00.534428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.534398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca" pod "6c2ffa87-b809-41d2-bea3-6e870f3e9969" (UID: "6c2ffa87-b809-41d2-bea3-6e870f3e9969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:42:00.534899 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.534868    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:00.536394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.536374    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6" pod "6c2ffa87-b809-41d2-bea3-6e870f3e9969" (UID: "6c2ffa87-b809-41d2-bea3-6e870f3e9969"). InnerVolumeSpecName "kube-api-access-qprs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:42:00.536479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.536403    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca" pod "6c2ffa87-b809-41d2-bea3-6e870f3e9969" (UID: "6c2ffa87-b809-41d2-bea3-6e870f3e9969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:42:00.635559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.635479    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2ffa87-b809-41d2-bea3-6e870f3e9969-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:00.635559 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:00.635519    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprs6\" (UniqueName: \"kubernetes.io/projected/6c2ffa87-b809-41d2-bea3-6e870f3e9969-kube-api-access-qprs6\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:01.312802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:01.312744    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"
Nov 01 20:42:01.312998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:01.312798    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74" event={"ID":"6c2ffa87-b809-41d2-bea3-6e870f3e9969","Type":"ContainerDied","Data":"40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a"}
Nov 01 20:42:01.312998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:01.312862    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e76c942e9b646495c7d3a80506606b21730451d54d1379ea9aac31e202b67a"
Nov 01 20:42:19.656060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.656023    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"]
Nov 01 20:42:19.656591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.656536    5041 cpu_manager.go:414] "containerMap: removing container" podUID="6c2ffa87-b809-41d2-bea3-6e870f3e9969" containerName="revoke-keys"
Nov 01 20:42:19.656591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.656548    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2ffa87-b809-41d2-bea3-6e870f3e9969" containerName="revoke-keys"
Nov 01 20:42:19.656739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.656624    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2ffa87-b809-41d2-bea3-6e870f3e9969" containerName="revoke-keys"
Nov 01 20:42:19.660168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.660145    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.664942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.664917    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:42:19.665267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.665038    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:42:19.665267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.665081    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:42:19.665462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.665217    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:42:19.666199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.666179    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"]
Nov 01 20:42:19.742670 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.742641    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.742872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.742694    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxz4\" (UniqueName: \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.742872 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.742847    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.843672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.843643    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.843908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.843697    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.843908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.843733    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxz4\" (UniqueName: \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.844371 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.844345    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.846071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.846047    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.851466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.851440    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxz4\" (UniqueName: \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\") pod \"maas-api-revoke-sub-f03629db-cj5kg\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:19.978450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:19.978367    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:20.253326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:20.253286    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"]
Nov 01 20:42:20.254553 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:42:20.254517    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35de46f_4fc4_443c_990f_75e1e0d1d843.slice/crio-b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1 WatchSource:0}: Error finding container b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1: Status 404 returned error can't find the container with id b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1
Nov 01 20:42:20.452577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:20.452541    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg" event={"ID":"b35de46f-4fc4-443c-990f-75e1e0d1d843","Type":"ContainerStarted","Data":"b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1"}
Nov 01 20:42:21.461907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:21.461824    5041 generic.go:356] "Generic (PLEG): container finished" podID="b35de46f-4fc4-443c-990f-75e1e0d1d843" containerID="e7a9a61fd2f2d3005c4af633c9c4f36043e316a8e6d4b255de683b4a48820578" exitCode=0
Nov 01 20:42:21.462317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:21.461901    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg" event={"ID":"b35de46f-4fc4-443c-990f-75e1e0d1d843","Type":"ContainerDied","Data":"e7a9a61fd2f2d3005c4af633c9c4f36043e316a8e6d4b255de683b4a48820578"}
Nov 01 20:42:22.612056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.612033    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:22.672863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.672834    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\") pod \"b35de46f-4fc4-443c-990f-75e1e0d1d843\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") "
Nov 01 20:42:22.673013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.672878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\" (UniqueName: \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\") pod \"b35de46f-4fc4-443c-990f-75e1e0d1d843\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") "
Nov 01 20:42:22.673013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.672926    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\") pod \"b35de46f-4fc4-443c-990f-75e1e0d1d843\" (UID: \"b35de46f-4fc4-443c-990f-75e1e0d1d843\") "
Nov 01 20:42:22.673357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.673333    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca" pod "b35de46f-4fc4-443c-990f-75e1e0d1d843" (UID: "b35de46f-4fc4-443c-990f-75e1e0d1d843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:42:22.674938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.674909    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca" pod "b35de46f-4fc4-443c-990f-75e1e0d1d843" (UID: "b35de46f-4fc4-443c-990f-75e1e0d1d843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:42:22.675053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.675002    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4" pod "b35de46f-4fc4-443c-990f-75e1e0d1d843" (UID: "b35de46f-4fc4-443c-990f-75e1e0d1d843"). InnerVolumeSpecName "kube-api-access-lhxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:42:22.773886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.773798    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:22.773886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.773842    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxz4\" (UniqueName: \"kubernetes.io/projected/b35de46f-4fc4-443c-990f-75e1e0d1d843-kube-api-access-lhxz4\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:22.773886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:22.773857    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b35de46f-4fc4-443c-990f-75e1e0d1d843-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:23.474395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:23.474364    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"
Nov 01 20:42:23.474573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:23.474396    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg" event={"ID":"b35de46f-4fc4-443c-990f-75e1e0d1d843","Type":"ContainerDied","Data":"b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1"}
Nov 01 20:42:23.474573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:23.474451    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0f4973ad231cf180a9e8ba05b10647ec872173cb0018174bf2d1ae83a7219d1"
Nov 01 20:42:28.540944 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:42:28.540903    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:42:35.345039 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.345003    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"]
Nov 01 20:42:35.345501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.345487    5041 cpu_manager.go:414] "containerMap: removing container" podUID="b35de46f-4fc4-443c-990f-75e1e0d1d843" containerName="revoke-keys"
Nov 01 20:42:35.345547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.345501    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35de46f-4fc4-443c-990f-75e1e0d1d843" containerName="revoke-keys"
Nov 01 20:42:35.345588 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.345574    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35de46f-4fc4-443c-990f-75e1e0d1d843" containerName="revoke-keys"
Nov 01 20:42:35.348929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.348913    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.352650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.352625    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:42:35.354437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.353835    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:42:35.354437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.353894    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:42:35.354437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.353955    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:42:35.355892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.355866    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"]
Nov 01 20:42:35.528665 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.528628    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tt4k\" (UniqueName: \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.528912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.528837    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.528912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.528878    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.630057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.629960    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.630057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.630013    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.630265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.630076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tt4k\" (UniqueName: \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.630738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.630717    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.632425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.632402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.637490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.637466    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tt4k\" (UniqueName: \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\") pod \"maas-api-revoke-sub-0165d7d2-slc92\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.666114 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.666091    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:35.944300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:35.944265    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"]
Nov 01 20:42:35.944861 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:42:35.944821    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0430059a_6cbb_4309_9aee_49aaa49691ed.slice/crio-728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3 WatchSource:0}: Error finding container 728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3: Status 404 returned error can't find the container with id 728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3
Nov 01 20:42:36.570721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:36.570681    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92" event={"ID":"0430059a-6cbb-4309-9aee-49aaa49691ed","Type":"ContainerStarted","Data":"947e334cde5aad53fd8a41046fed48e7622274ac8b007ea780fe4a1af3846b36"}
Nov 01 20:42:36.571243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:36.570763    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92" event={"ID":"0430059a-6cbb-4309-9aee-49aaa49691ed","Type":"ContainerStarted","Data":"728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3"}
Nov 01 20:42:36.586674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:36.586623    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92" podStartSLOduration=1.5866115729999999 podStartE2EDuration="1.586611573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:42:36.585527417 +0000 UTC m=+2644.306606094" watchObservedRunningTime="2026-11-01 20:42:36.586611573 +0000 UTC m=+2644.307690248"
Nov 01 20:42:37.264830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:37.264799    5041 scope.go:122] "RemoveContainer" containerID="7c26fcc098282ce2a3883c0d3bfc4f308879b957c48c509a9f9c625a900e3e14"
Nov 01 20:42:37.578299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:37.578261    5041 generic.go:356] "Generic (PLEG): container finished" podID="0430059a-6cbb-4309-9aee-49aaa49691ed" containerID="947e334cde5aad53fd8a41046fed48e7622274ac8b007ea780fe4a1af3846b36" exitCode=0
Nov 01 20:42:37.578698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:37.578346    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92" event={"ID":"0430059a-6cbb-4309-9aee-49aaa49691ed","Type":"ContainerDied","Data":"947e334cde5aad53fd8a41046fed48e7622274ac8b007ea780fe4a1af3846b36"}
Nov 01 20:42:38.730833 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.730811    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:38.874638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.874546    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\" (UniqueName: \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\") pod \"0430059a-6cbb-4309-9aee-49aaa49691ed\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") "
Nov 01 20:42:38.874864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.874673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\") pod \"0430059a-6cbb-4309-9aee-49aaa49691ed\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") "
Nov 01 20:42:38.874864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.874713    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\") pod \"0430059a-6cbb-4309-9aee-49aaa49691ed\" (UID: \"0430059a-6cbb-4309-9aee-49aaa49691ed\") "
Nov 01 20:42:38.875030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.875005    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca" pod "0430059a-6cbb-4309-9aee-49aaa49691ed" (UID: "0430059a-6cbb-4309-9aee-49aaa49691ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:42:38.875141 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.875122    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:38.876920 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.876892    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k" pod "0430059a-6cbb-4309-9aee-49aaa49691ed" (UID: "0430059a-6cbb-4309-9aee-49aaa49691ed"). InnerVolumeSpecName "kube-api-access-8tt4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:42:38.877032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.876994    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca" pod "0430059a-6cbb-4309-9aee-49aaa49691ed" (UID: "0430059a-6cbb-4309-9aee-49aaa49691ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:42:38.976610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.976572    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0430059a-6cbb-4309-9aee-49aaa49691ed-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:38.976610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:38.976602    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tt4k\" (UniqueName: \"kubernetes.io/projected/0430059a-6cbb-4309-9aee-49aaa49691ed-kube-api-access-8tt4k\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:39.591904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:39.591871    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"
Nov 01 20:42:39.592092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:39.591902    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92" event={"ID":"0430059a-6cbb-4309-9aee-49aaa49691ed","Type":"ContainerDied","Data":"728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3"}
Nov 01 20:42:39.592092 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:39.591952    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="728d0804aff70708adf41f4fe4193aba60d88ef482d63f87158b798919e2a0b3"
Nov 01 20:42:46.040479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.040399    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"]
Nov 01 20:42:46.040957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.040941    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0430059a-6cbb-4309-9aee-49aaa49691ed" containerName="revoke-keys"
Nov 01 20:42:46.041014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.040958    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0430059a-6cbb-4309-9aee-49aaa49691ed" containerName="revoke-keys"
Nov 01 20:42:46.041059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.041048    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0430059a-6cbb-4309-9aee-49aaa49691ed" containerName="revoke-keys"
Nov 01 20:42:46.044490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.044469    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.048030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.048002    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:42:46.049925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.049903    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:42:46.050085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.049901    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:42:46.050164 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.049903    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:42:46.052474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.052451    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"]
Nov 01 20:42:46.131748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.131713    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.131929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.131804    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9kr\" (UniqueName: \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.131929 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.131880    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.233042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.233001    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.233228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.233078    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9kr\" (UniqueName: \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.233228 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.233102    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.233771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.233752    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.235434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.235414    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.240679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.240656    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9kr\" (UniqueName: \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\") pod \"maas-api-revoke-sub-6b19338f-gbz92\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.371580 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.371497    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:46.647898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:46.647872    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"]
Nov 01 20:42:46.649369 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:42:46.649339    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f59b4e_4981_4cd2_acf7_719fd8d105af.slice/crio-5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84 WatchSource:0}: Error finding container 5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84: Status 404 returned error can't find the container with id 5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84
Nov 01 20:42:47.642743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:47.642702    5041 generic.go:356] "Generic (PLEG): container finished" podID="55f59b4e-4981-4cd2-acf7-719fd8d105af" containerID="ce2921a80b653f8ce1421d4d3b4630f7151d1bb5dcc8cf129e57406a7834ab48" exitCode=0
Nov 01 20:42:47.643254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:47.642807    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92" event={"ID":"55f59b4e-4981-4cd2-acf7-719fd8d105af","Type":"ContainerDied","Data":"ce2921a80b653f8ce1421d4d3b4630f7151d1bb5dcc8cf129e57406a7834ab48"}
Nov 01 20:42:47.643254 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:47.642857    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92" event={"ID":"55f59b4e-4981-4cd2-acf7-719fd8d105af","Type":"ContainerStarted","Data":"5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84"}
Nov 01 20:42:48.792298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.792267    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:48.857222 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.857188    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\") pod \"55f59b4e-4981-4cd2-acf7-719fd8d105af\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") "
Nov 01 20:42:48.857403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.857323    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\" (UniqueName: \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\") pod \"55f59b4e-4981-4cd2-acf7-719fd8d105af\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") "
Nov 01 20:42:48.857403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.857351    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\") pod \"55f59b4e-4981-4cd2-acf7-719fd8d105af\" (UID: \"55f59b4e-4981-4cd2-acf7-719fd8d105af\") "
Nov 01 20:42:48.857739 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.857716    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca" pod "55f59b4e-4981-4cd2-acf7-719fd8d105af" (UID: "55f59b4e-4981-4cd2-acf7-719fd8d105af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:42:48.859309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.859282    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca" pod "55f59b4e-4981-4cd2-acf7-719fd8d105af" (UID: "55f59b4e-4981-4cd2-acf7-719fd8d105af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:42:48.859474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.859450    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr" pod "55f59b4e-4981-4cd2-acf7-719fd8d105af" (UID: "55f59b4e-4981-4cd2-acf7-719fd8d105af"). InnerVolumeSpecName "kube-api-access-zp9kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:42:48.958198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.958117    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp9kr\" (UniqueName: \"kubernetes.io/projected/55f59b4e-4981-4cd2-acf7-719fd8d105af-kube-api-access-zp9kr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:48.958198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.958146    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:48.958198 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:48.958160    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f59b4e-4981-4cd2-acf7-719fd8d105af-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:42:49.656869 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:49.656826    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92" event={"ID":"55f59b4e-4981-4cd2-acf7-719fd8d105af","Type":"ContainerDied","Data":"5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84"}
Nov 01 20:42:49.657028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:49.656906    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5820050f7ce5f990250e2dfde2c78f6252fce5c761d8073fb393d1693dabcc84"
Nov 01 20:42:49.657028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:42:49.656836    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"
Nov 01 20:42:58.614539 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:42:58.614495    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:43:26.021300 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:26.021267    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"]
Nov 01 20:43:26.024334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:26.024312    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117cdddd-f6v2t"]
Nov 01 20:43:26.427927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:26.427894    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44c282e-3c2a-4e94-b296-b02aed12446c" path="/var/lib/kubelet/pods/d44c282e-3c2a-4e94-b296-b02aed12446c/volumes"
Nov 01 20:43:28.540628 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:43:28.540588    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:43:33.042596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:33.042564    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:43:33.055440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:33.055416    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:43:36.021814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:36.021766    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"]
Nov 01 20:43:36.024667 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:36.024626    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef9206f2-6grg4"]
Nov 01 20:43:36.430011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:36.428423    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d37387-559b-4619-a269-51bad362e9a4" path="/var/lib/kubelet/pods/84d37387-559b-4619-a269-51bad362e9a4/volumes"
Nov 01 20:43:46.020427 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:46.020390    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"]
Nov 01 20:43:46.024324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:46.024300    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fd24d96-ktf2v"]
Nov 01 20:43:46.427317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:46.427280    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ce6510-f58c-42fa-8fbd-19dcc202b3c4" path="/var/lib/kubelet/pods/e9ce6510-f58c-42fa-8fbd-19dcc202b3c4/volumes"
Nov 01 20:43:57.021386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:57.021352    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"]
Nov 01 20:43:57.024160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:57.024139    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18977da6-9587j"]
Nov 01 20:43:58.427018 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:43:58.426986    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea404f3-41d9-4955-b8bb-385190f6ac3a" path="/var/lib/kubelet/pods/0ea404f3-41d9-4955-b8bb-385190f6ac3a/volumes"
Nov 01 20:43:58.540407 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:43:58.540370    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:44:01.247319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.247282    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"]
Nov 01 20:44:01.247842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.247825    5041 cpu_manager.go:414] "containerMap: removing container" podUID="55f59b4e-4981-4cd2-acf7-719fd8d105af" containerName="revoke-keys"
Nov 01 20:44:01.247897 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.247844    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f59b4e-4981-4cd2-acf7-719fd8d105af" containerName="revoke-keys"
Nov 01 20:44:01.247971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.247961    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f59b4e-4981-4cd2-acf7-719fd8d105af" containerName="revoke-keys"
Nov 01 20:44:01.251181 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.251165    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.256343 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.256296    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:44:01.256925 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.256901    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:44:01.258191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.258166    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"]
Nov 01 20:44:01.282666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.282624    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.282882 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.282827    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.282991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.282948    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnpd\" (UniqueName: \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.292237 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.292217    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:44:01.292330 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.292313    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:44:01.384286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.384254    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.384443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.384360    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.384443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.384439    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnpd\" (UniqueName: \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.385060 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.385039    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.386751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.386726    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.392149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.392129    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnpd\" (UniqueName: \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\") pod \"maas-api-revoke-sub-c4db39d7-fw59q\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.571027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.570991    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:01.851839 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:01.846006    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"]
Nov 01 20:44:01.851839 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:44:01.846036    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d29371_06bf_49b8_8355_342b47b71acf.slice/crio-ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3 WatchSource:0}: Error finding container ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3: Status 404 returned error can't find the container with id ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3
Nov 01 20:44:02.194010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:02.193971    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q" event={"ID":"10d29371-06bf-49b8-8355-342b47b71acf","Type":"ContainerStarted","Data":"eadcc1f328db762f7f919f3a69c4576dadd9fdcf549df2187af71f86bbfc2592"}
Nov 01 20:44:02.194213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:02.194027    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q" event={"ID":"10d29371-06bf-49b8-8355-342b47b71acf","Type":"ContainerStarted","Data":"ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3"}
Nov 01 20:44:02.211261 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:02.211210    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q" podStartSLOduration=1.211192449 podStartE2EDuration="1.211192449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:02.208346902 +0000 UTC m=+2729.929425590" watchObservedRunningTime="2026-11-01 20:44:02.211192449 +0000 UTC m=+2729.932271125"
Nov 01 20:44:03.201043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:03.201009    5041 generic.go:356] "Generic (PLEG): container finished" podID="10d29371-06bf-49b8-8355-342b47b71acf" containerID="eadcc1f328db762f7f919f3a69c4576dadd9fdcf549df2187af71f86bbfc2592" exitCode=0
Nov 01 20:44:03.201437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:03.201089    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q" event={"ID":"10d29371-06bf-49b8-8355-342b47b71acf","Type":"ContainerDied","Data":"eadcc1f328db762f7f919f3a69c4576dadd9fdcf549df2187af71f86bbfc2592"}
Nov 01 20:44:04.364850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.364824    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:04.413140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.413106    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\") pod \"10d29371-06bf-49b8-8355-342b47b71acf\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") "
Nov 01 20:44:04.413337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.413185    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\") pod \"10d29371-06bf-49b8-8355-342b47b71acf\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") "
Nov 01 20:44:04.413337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.413244    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\" (UniqueName: \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\") pod \"10d29371-06bf-49b8-8355-342b47b71acf\" (UID: \"10d29371-06bf-49b8-8355-342b47b71acf\") "
Nov 01 20:44:04.413606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.413583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca" pod "10d29371-06bf-49b8-8355-342b47b71acf" (UID: "10d29371-06bf-49b8-8355-342b47b71acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:44:04.415388 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.415354    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd" pod "10d29371-06bf-49b8-8355-342b47b71acf" (UID: "10d29371-06bf-49b8-8355-342b47b71acf"). InnerVolumeSpecName "kube-api-access-7xnpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:44:04.415486 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.415361    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca" pod "10d29371-06bf-49b8-8355-342b47b71acf" (UID: "10d29371-06bf-49b8-8355-342b47b71acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:44:04.514459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.514362    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnpd\" (UniqueName: \"kubernetes.io/projected/10d29371-06bf-49b8-8355-342b47b71acf-kube-api-access-7xnpd\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:04.514459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.514399    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:04.514459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:04.514413    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d29371-06bf-49b8-8355-342b47b71acf-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:44:05.215275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:05.215243    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"
Nov 01 20:44:05.215456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:05.215273    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q" event={"ID":"10d29371-06bf-49b8-8355-342b47b71acf","Type":"ContainerDied","Data":"ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3"}
Nov 01 20:44:05.215456 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:05.215322    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec79d8960c4f566493619ec9aad2306ebacc586e409ee079f02c5fd85d4287c3"
Nov 01 20:44:07.020835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:07.020765    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"]
Nov 01 20:44:07.023677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:07.023636    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1d2fca-r777l"]
Nov 01 20:44:08.427504 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:08.427467    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c449f39-e24e-43ef-9357-e5e850a643c3" path="/var/lib/kubelet/pods/8c449f39-e24e-43ef-9357-e5e850a643c3/volumes"
Nov 01 20:44:15.824146 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.823921    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:44:15.825602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.825577    5041 cpu_manager.go:414] "containerMap: removing container" podUID="10d29371-06bf-49b8-8355-342b47b71acf" containerName="revoke-keys"
Nov 01 20:44:15.825804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.825762    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d29371-06bf-49b8-8355-342b47b71acf" containerName="revoke-keys"
Nov 01 20:44:15.826079 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.826062    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d29371-06bf-49b8-8355-342b47b71acf" containerName="revoke-keys"
Nov 01 20:44:15.836106 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.835382    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.837049 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.837021    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:44:15.839900 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.839883    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4cb35ada-kserve-self-signed-certs\""
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.935965    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.936059    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.936097    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.936166    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.936198    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:15.936418 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:15.936225    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lx4\" (UniqueName: \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037324    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037417    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037450    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037511    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037541    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.037737 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.037568    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lx4\" (UniqueName: \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.038036 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.038007    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.038233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.038062    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.038349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.038159    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.040805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.040227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.040805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.040406    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.064802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.057625    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lx4\" (UniqueName: \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\") pod \"e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.131957 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.131853    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:44:16.139968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.137361    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:44:16.139968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.137514    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.183841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.183815    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4cb35ada-epp-sa-dockercfg-tldzp\""
Nov 01 20:44:16.200483 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.199402    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:16.254513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.246529    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jc9\" (UniqueName: \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.254513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.246609    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.370807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.350436    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jc9\" (UniqueName: \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.370807 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.363928    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.397858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.382482    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.423825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.421515    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jc9\" (UniqueName: \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\") pod \"e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") " pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.590806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.565989    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:16.735156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:16.733592    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:44:17.064440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.064396    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:44:17.076511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.075886    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 20:44:17.346854 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.346817    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" event={"ID":"c1cdd583-1688-4439-940e-1d551e909038","Type":"ContainerStarted","Data":"5bf58155ccdf8a31d1667a6982b5911662f30730f51a6786701a1c7c0a7b0ff5"}
Nov 01 20:44:17.348417 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.348387    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" event={"ID":"4cf70269-9117-4cec-8d26-c238d676e518","Type":"ContainerStarted","Data":"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"}
Nov 01 20:44:17.348555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.348426    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" event={"ID":"4cf70269-9117-4cec-8d26-c238d676e518","Type":"ContainerStarted","Data":"71063f4121250c6b13a938ecdccc43a38c17419d29b34f92923c9a74ae70fa6a"}
Nov 01 20:44:17.365709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:17.365661    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" podStartSLOduration=2.3656481830000002 podStartE2EDuration="2.365648183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:17.363371254 +0000 UTC m=+2745.084449930" watchObservedRunningTime="2026-11-01 20:44:17.365648183 +0000 UTC m=+2745.086726856"
Nov 01 20:44:18.024406 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:18.024369    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"]
Nov 01 20:44:18.030573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:18.030547    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c431e2-clrz4"]
Nov 01 20:44:18.231724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:18.231687    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:18.428604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:18.428570    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5af236d-1015-4e9e-bb8b-8ecb3366e2f9" path="/var/lib/kubelet/pods/f5af236d-1015-4e9e-bb8b-8ecb3366e2f9/volumes"
Nov 01 20:44:20.412105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:20.412063    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" event={"ID":"c1cdd583-1688-4439-940e-1d551e909038","Type":"ContainerStarted","Data":"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"}
Nov 01 20:44:20.413553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:20.413530    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:20.432815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:20.430706    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podStartSLOduration=2.003760781 podStartE2EDuration="4.430692252s" totalImagesPullingTime="2.426931471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:44:20.430016302 +0000 UTC m=+2748.151094981" watchObservedRunningTime="2026-11-01 20:44:20.430692252 +0000 UTC m=+2748.151770928"
Nov 01 20:44:26.231612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:26.231571    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:26.244403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:26.244378    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:26.466462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:26.466429    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:44:28.019528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:28.019498    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"]
Nov 01 20:44:28.022210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:28.022189    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78e063dd-2qnz9"]
Nov 01 20:44:28.426584 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:28.426552    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41925c63-45cb-49e4-9e14-90b5f3802c2e" path="/var/lib/kubelet/pods/41925c63-45cb-49e4-9e14-90b5f3802c2e/volumes"
Nov 01 20:44:28.540541 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:44:28.540504    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:44:37.299004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:37.298966    5041 scope.go:122] "RemoveContainer" containerID="f736e45726db359f244c1b145c1712f086fa8d5f99dabf2deb606c13550936d4"
Nov 01 20:44:37.308707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:37.308681    5041 scope.go:122] "RemoveContainer" containerID="41e7a9330a2f4bf9e7c457aa7f5c6927ee949e392bb7772e80f30949f76efa63"
Nov 01 20:44:38.021326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:38.021293    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"]
Nov 01 20:44:38.023865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:38.023843    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4789b905-v6mg8"]
Nov 01 20:44:38.430979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:38.428839    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160c6aa8-f76f-4728-be81-defa54c81326" path="/var/lib/kubelet/pods/160c6aa8-f76f-4728-be81-defa54c81326/volumes"
Nov 01 20:44:48.020525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:48.020486    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"]
Nov 01 20:44:48.023030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:48.023008    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01a87047-prfhj"]
Nov 01 20:44:48.427750 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:48.427717    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6fa16c-a993-481c-a898-eee2c75cc673" path="/var/lib/kubelet/pods/8a6fa16c-a993-481c-a898-eee2c75cc673/volumes"
Nov 01 20:44:52.433010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:52.432981    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:44:58.541088 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:44:58.541051    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:44:59.020841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:59.020703    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"]
Nov 01 20:44:59.023533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:44:59.023500    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4158a440-w84tt"]
Nov 01 20:45:00.428558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:00.428524    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76201f1e-718a-49d2-ac98-92d109b0961f" path="/var/lib/kubelet/pods/76201f1e-718a-49d2-ac98-92d109b0961f/volumes"
Nov 01 20:45:06.131314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.131277    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"]
Nov 01 20:45:06.135830 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.135795    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.139814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.139754    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:45:06.139962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.139820    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:45:06.140055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.139980    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:45:06.140372 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.140352    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:45:06.142433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.142405    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"]
Nov 01 20:45:06.210497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.210462    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.210680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.210602    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.210680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.210625    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn4b\" (UniqueName: \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.311964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.311930    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.312172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.311971    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn4b\" (UniqueName: \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.312172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.312011    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.312656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.312635    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.314470 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.314445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.319986 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.319962    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn4b\" (UniqueName: \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\") pod \"maas-api-revoke-sub-b48d306b-pzbxl\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.456610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.456524    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:06.731026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.730997    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"]
Nov 01 20:45:06.732679 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:45:06.732653    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f41ab52_7a07_488a_93aa_1482fba8c794.slice/crio-815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d WatchSource:0}: Error finding container 815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d: Status 404 returned error can't find the container with id 815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d
Nov 01 20:45:06.747764 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:06.747738    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl" event={"ID":"0f41ab52-7a07-488a-93aa-1482fba8c794","Type":"ContainerStarted","Data":"815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d"}
Nov 01 20:45:07.755982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:07.755946    5041 generic.go:356] "Generic (PLEG): container finished" podID="0f41ab52-7a07-488a-93aa-1482fba8c794" containerID="1257dc7f7c4947db392df389ea8e98c675fb069b905d34d3ee939d7d7cb2ea26" exitCode=0
Nov 01 20:45:07.756370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:07.756006    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl" event={"ID":"0f41ab52-7a07-488a-93aa-1482fba8c794","Type":"ContainerDied","Data":"1257dc7f7c4947db392df389ea8e98c675fb069b905d34d3ee939d7d7cb2ea26"}
Nov 01 20:45:08.915067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:08.915041    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:09.020809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.020700    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"]
Nov 01 20:45:09.023478 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.023451    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff92358e-9lpvm"]
Nov 01 20:45:09.039042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.039013    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\" (UniqueName: \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\") pod \"0f41ab52-7a07-488a-93aa-1482fba8c794\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") "
Nov 01 20:45:09.039179 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.039061    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\") pod \"0f41ab52-7a07-488a-93aa-1482fba8c794\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") "
Nov 01 20:45:09.039243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.039179    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\") pod \"0f41ab52-7a07-488a-93aa-1482fba8c794\" (UID: \"0f41ab52-7a07-488a-93aa-1482fba8c794\") "
Nov 01 20:45:09.039532 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.039508    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca" pod "0f41ab52-7a07-488a-93aa-1482fba8c794" (UID: "0f41ab52-7a07-488a-93aa-1482fba8c794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:45:09.041199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.041179    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca" pod "0f41ab52-7a07-488a-93aa-1482fba8c794" (UID: "0f41ab52-7a07-488a-93aa-1482fba8c794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:45:09.041279 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.041264    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b" pod "0f41ab52-7a07-488a-93aa-1482fba8c794" (UID: "0f41ab52-7a07-488a-93aa-1482fba8c794"). InnerVolumeSpecName "kube-api-access-csn4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:09.140726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.140689    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csn4b\" (UniqueName: \"kubernetes.io/projected/0f41ab52-7a07-488a-93aa-1482fba8c794-kube-api-access-csn4b\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.140726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.140722    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.140726 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.140733    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f41ab52-7a07-488a-93aa-1482fba8c794-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:09.770227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.770193    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"
Nov 01 20:45:09.770414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.770228    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl" event={"ID":"0f41ab52-7a07-488a-93aa-1482fba8c794","Type":"ContainerDied","Data":"815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d"}
Nov 01 20:45:09.770414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:09.770279    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815991b9322262a5b220eee128ea48ab268dfb8a36535b3d6091725faec8f52d"
Nov 01 20:45:10.427896 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:10.427862    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114f165d-d59f-4c88-9581-af2e8d2aa53f" path="/var/lib/kubelet/pods/114f165d-d59f-4c88-9581-af2e8d2aa53f/volumes"
Nov 01 20:45:17.441493 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:17.441453    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:45:17.447829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:17.447799    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:45:19.019344 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:19.019302    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"]
Nov 01 20:45:19.023432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:19.023406    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89db3ea8-wtm9r"]
Nov 01 20:45:20.439333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:20.439288    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f128807e-d623-4986-9e4f-def79e80a71a" path="/var/lib/kubelet/pods/f128807e-d623-4986-9e4f-def79e80a71a/volumes"
Nov 01 20:45:22.433071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:22.433014    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:28.542000 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:45:28.541963    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:45:32.020525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.020484    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"]
Nov 01 20:45:32.023303 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.023273    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d126ad-v9smk"]
Nov 01 20:45:32.427696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.427660    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a8718e-7be0-44d4-a4bd-e0113cca2201" path="/var/lib/kubelet/pods/20a8718e-7be0-44d4-a4bd-e0113cca2201/volumes"
Nov 01 20:45:32.433176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.433145    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:32.468867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.468824    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" podUID="4cf70269-9117-4cec-8d26-c238d676e518" containerName="main" containerID="cri-o://d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77" gracePeriod=45
Nov 01 20:45:32.470208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.470182    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" containerID="cri-o://f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f" gracePeriod=45
Nov 01 20:45:32.747619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.747593    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:45:32.801210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801171    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801210 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801212    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\" (UniqueName: \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801248    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801323    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801357    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801455    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\") pod \"4cf70269-9117-4cec-8d26-c238d676e518\" (UID: \"4cf70269-9117-4cec-8d26-c238d676e518\") "
Nov 01 20:45:32.801507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801491    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:32.801823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801539    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:32.801823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801758    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:32.801952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801834    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.801952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.801848    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.803796 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.803744    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:45:32.804035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.804016    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "kube-api-access-78lx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:45:32.804137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.804114    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs" pod "4cf70269-9117-4cec-8d26-c238d676e518" (UID: "4cf70269-9117-4cec-8d26-c238d676e518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:45:32.902728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.902689    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lx4\" (UniqueName: \"kubernetes.io/projected/4cf70269-9117-4cec-8d26-c238d676e518-kube-api-access-78lx4\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.902728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.902722    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf70269-9117-4cec-8d26-c238d676e518-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.902728 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.902736    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.903027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.902751    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf70269-9117-4cec-8d26-c238d676e518-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:45:32.976045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.975954    5041 generic.go:356] "Generic (PLEG): container finished" podID="4cf70269-9117-4cec-8d26-c238d676e518" containerID="d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77" exitCode=0
Nov 01 20:45:32.976045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.976003    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"
Nov 01 20:45:32.976045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.976026    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" event={"ID":"4cf70269-9117-4cec-8d26-c238d676e518","Type":"ContainerDied","Data":"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"}
Nov 01 20:45:32.976306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.976061    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k" event={"ID":"4cf70269-9117-4cec-8d26-c238d676e518","Type":"ContainerDied","Data":"71063f4121250c6b13a938ecdccc43a38c17419d29b34f92923c9a74ae70fa6a"}
Nov 01 20:45:32.976306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.976077    5041 scope.go:122] "RemoveContainer" containerID="d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"
Nov 01 20:45:32.988903 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.988876    5041 scope.go:122] "RemoveContainer" containerID="d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"
Nov 01 20:45:32.989766 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:45:32.989736    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77\": container with ID starting with d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77 not found: ID does not exist" containerID="d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"
Nov 01 20:45:32.989939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.989806    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77"} err="failed to get container status \"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77\": rpc error: code = NotFound desc = could not find container \"d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77\": container with ID starting with d5f5a005ec37f222e7dd8ce3e2abd340a3c23e34ef6461cfb1c0697deb39ed77 not found: ID does not exist"
Nov 01 20:45:32.999107 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:32.999078    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:45:33.001595 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:33.001571    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-5855f876c6-knm8k"]
Nov 01 20:45:34.427062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:34.427020    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf70269-9117-4cec-8d26-c238d676e518" path="/var/lib/kubelet/pods/4cf70269-9117-4cec-8d26-c238d676e518/volumes"
Nov 01 20:45:37.340172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.340141    5041 scope.go:122] "RemoveContainer" containerID="d2ca1174460a54f71ae6a672223c8c9f76ce580f31572e186265a7f35c73329d"
Nov 01 20:45:37.349687 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.349666    5041 scope.go:122] "RemoveContainer" containerID="3fdb13241b62ba27148dd495a23fec7938a6153cd285b9395104cd9ce178fe5c"
Nov 01 20:45:37.358772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.358742    5041 scope.go:122] "RemoveContainer" containerID="5d964d332597112698135779b89239ca803beed0b820dcc626635fba659f7483"
Nov 01 20:45:37.367297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.367274    5041 scope.go:122] "RemoveContainer" containerID="2cf3166e52faa3ac8b8c2d0cf71f3704b45498bf2d5d09ff481686b52974160e"
Nov 01 20:45:37.375793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.375762    5041 scope.go:122] "RemoveContainer" containerID="98f1db3bb38ebbbb61f82398c9b652e6a5a68b8571de055309c431d19c32f795"
Nov 01 20:45:37.384192 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:37.384174    5041 scope.go:122] "RemoveContainer" containerID="0c2d0446adfa7e13779a122a32fc51b342de3734097766ad9d1ae3e3410ce68e"
Nov 01 20:45:42.432771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:42.432733    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:42.433281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:42.432853    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:45:52.020432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:52.020395    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"]
Nov 01 20:45:52.026030 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:52.026001    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683fc09a-9jdsf"]
Nov 01 20:45:52.427282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:52.427244    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059f4463-d71a-437b-b495-7b7baba3b061" path="/var/lib/kubelet/pods/059f4463-d71a-437b-b495-7b7baba3b061/volumes"
Nov 01 20:45:52.432413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:45:52.432376    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:45:58.545035 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:45:58.544995    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:46:02.432919 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.432877    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 20:46:02.742119 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.742087    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:46:02.771907 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.771874    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\" (UniqueName: \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\") pod \"c1cdd583-1688-4439-940e-1d551e909038\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") "
Nov 01 20:46:02.772081 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.771989    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\") pod \"c1cdd583-1688-4439-940e-1d551e909038\" (UID: \"c1cdd583-1688-4439-940e-1d551e909038\") "
Nov 01 20:46:02.774145 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.774106    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9" pod "c1cdd583-1688-4439-940e-1d551e909038" (UID: "c1cdd583-1688-4439-940e-1d551e909038"). InnerVolumeSpecName "kube-api-access-f6jc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:02.774290 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.774158    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs" pod "c1cdd583-1688-4439-940e-1d551e909038" (UID: "c1cdd583-1688-4439-940e-1d551e909038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:02.873284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.873249    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jc9\" (UniqueName: \"kubernetes.io/projected/c1cdd583-1688-4439-940e-1d551e909038-kube-api-access-f6jc9\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:02.873284 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:02.873277    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1cdd583-1688-4439-940e-1d551e909038-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:03.167791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.167746    5041 generic.go:356] "Generic (PLEG): container finished" podID="c1cdd583-1688-4439-940e-1d551e909038" containerID="f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f" exitCode=0
Nov 01 20:46:03.168019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.167804    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"
Nov 01 20:46:03.168019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.167818    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" event={"ID":"c1cdd583-1688-4439-940e-1d551e909038","Type":"ContainerDied","Data":"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"}
Nov 01 20:46:03.168019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.167886    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf" event={"ID":"c1cdd583-1688-4439-940e-1d551e909038","Type":"ContainerDied","Data":"5bf58155ccdf8a31d1667a6982b5911662f30730f51a6786701a1c7c0a7b0ff5"}
Nov 01 20:46:03.168019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.167915    5041 scope.go:122] "RemoveContainer" containerID="f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"
Nov 01 20:46:03.178496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.178474    5041 scope.go:122] "RemoveContainer" containerID="f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"
Nov 01 20:46:03.178829 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:46:03.178771    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f\": container with ID starting with f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f not found: ID does not exist" containerID="f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"
Nov 01 20:46:03.178935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.178839    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f"} err="failed to get container status \"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f\": rpc error: code = NotFound desc = could not find container \"f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f\": container with ID starting with f667bc1a14e52393787f3488151880304406c21c80459cca11cf9b426474c02f not found: ID does not exist"
Nov 01 20:46:03.198016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.197972    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:46:03.198179 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:03.198047    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4cb35ada-kserve-router-scheduler-7b96567c4c-8n5mf"]
Nov 01 20:46:04.427095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:04.427062    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cdd583-1688-4439-940e-1d551e909038" path="/var/lib/kubelet/pods/c1cdd583-1688-4439-940e-1d551e909038/volumes"
Nov 01 20:46:09.921791 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.921751    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"]
Nov 01 20:46:09.922306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922289    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main"
Nov 01 20:46:09.922386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922308    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main"
Nov 01 20:46:09.922386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922328    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0f41ab52-7a07-488a-93aa-1482fba8c794" containerName="revoke-keys"
Nov 01 20:46:09.922386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922333    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f41ab52-7a07-488a-93aa-1482fba8c794" containerName="revoke-keys"
Nov 01 20:46:09.922386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922350    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4cf70269-9117-4cec-8d26-c238d676e518" containerName="main"
Nov 01 20:46:09.922386 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922356    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf70269-9117-4cec-8d26-c238d676e518" containerName="main"
Nov 01 20:46:09.922571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922441    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cdd583-1688-4439-940e-1d551e909038" containerName="main"
Nov 01 20:46:09.922571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922451    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f41ab52-7a07-488a-93aa-1482fba8c794" containerName="revoke-keys"
Nov 01 20:46:09.922571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.922460    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf70269-9117-4cec-8d26-c238d676e518" containerName="main"
Nov 01 20:46:09.927158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.927139    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:09.933265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.933236    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"]
Nov 01 20:46:09.934352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.934332    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:46:09.934352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.934350    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:46:09.934524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.934371    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:46:09.934524 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:09.934374    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:46:10.048009 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.047966    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.048212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.048139    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhkn\" (UniqueName: \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.048212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.048176    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.149528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.149486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhkn\" (UniqueName: \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.149528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.149529    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.149745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.149610    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.150387 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.150361    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.152255 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.152229    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.157685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.157662    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhkn\" (UniqueName: \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\") pod \"maas-api-revoke-sub-452b6d7b-6fvxp\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.246970 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.246884    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:10.520252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:10.520230    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"]
Nov 01 20:46:10.521715 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:46:10.521685    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d39646_f793_4e6c_a228_3bbe22942190.slice/crio-aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6 WatchSource:0}: Error finding container aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6: Status 404 returned error can't find the container with id aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6
Nov 01 20:46:11.229803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:11.229679    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp" event={"ID":"72d39646-f793-4e6c-a228-3bbe22942190","Type":"ContainerStarted","Data":"4638da3a259d0f8f0eb9a1d7819894a87b0e4d2c3daa78de065479a44d822f49"}
Nov 01 20:46:11.230301 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:11.229811    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp" event={"ID":"72d39646-f793-4e6c-a228-3bbe22942190","Type":"ContainerStarted","Data":"aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6"}
Nov 01 20:46:11.246678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:11.246619    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp" podStartSLOduration=2.24660547 podStartE2EDuration="2.24660547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:11.244808636 +0000 UTC m=+2858.965887314" watchObservedRunningTime="2026-11-01 20:46:11.24660547 +0000 UTC m=+2858.967684145"
Nov 01 20:46:12.237224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:12.237187    5041 generic.go:356] "Generic (PLEG): container finished" podID="72d39646-f793-4e6c-a228-3bbe22942190" containerID="4638da3a259d0f8f0eb9a1d7819894a87b0e4d2c3daa78de065479a44d822f49" exitCode=0
Nov 01 20:46:12.237676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:12.237249    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp" event={"ID":"72d39646-f793-4e6c-a228-3bbe22942190","Type":"ContainerDied","Data":"4638da3a259d0f8f0eb9a1d7819894a87b0e4d2c3daa78de065479a44d822f49"}
Nov 01 20:46:13.653602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.653258    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:13.721879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.721552    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\") pod \"72d39646-f793-4e6c-a228-3bbe22942190\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") "
Nov 01 20:46:13.721879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.721639    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\" (UniqueName: \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\") pod \"72d39646-f793-4e6c-a228-3bbe22942190\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") "
Nov 01 20:46:13.727815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.725876    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\") pod \"72d39646-f793-4e6c-a228-3bbe22942190\" (UID: \"72d39646-f793-4e6c-a228-3bbe22942190\") "
Nov 01 20:46:13.727815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.726008    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca" pod "72d39646-f793-4e6c-a228-3bbe22942190" (UID: "72d39646-f793-4e6c-a228-3bbe22942190"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:46:13.727815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.726274    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:13.729419 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.728313    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca" pod "72d39646-f793-4e6c-a228-3bbe22942190" (UID: "72d39646-f793-4e6c-a228-3bbe22942190"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:13.734939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.734901    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn" pod "72d39646-f793-4e6c-a228-3bbe22942190" (UID: "72d39646-f793-4e6c-a228-3bbe22942190"). InnerVolumeSpecName "kube-api-access-mbhkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:13.827377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.827340    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72d39646-f793-4e6c-a228-3bbe22942190-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:13.827377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:13.827377    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhkn\" (UniqueName: \"kubernetes.io/projected/72d39646-f793-4e6c-a228-3bbe22942190-kube-api-access-mbhkn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:14.260252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:14.260214    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp" event={"ID":"72d39646-f793-4e6c-a228-3bbe22942190","Type":"ContainerDied","Data":"aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6"}
Nov 01 20:46:14.260443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:14.260266    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aac6852d237d0486f18d9a206247ababa668ca4c2afeea364ceede56315ff2a6"
Nov 01 20:46:14.260443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:14.260233    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"
Nov 01 20:46:25.777695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.777662    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"]
Nov 01 20:46:25.778336 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.778318    5041 cpu_manager.go:414] "containerMap: removing container" podUID="72d39646-f793-4e6c-a228-3bbe22942190" containerName="revoke-keys"
Nov 01 20:46:25.778400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.778337    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d39646-f793-4e6c-a228-3bbe22942190" containerName="revoke-keys"
Nov 01 20:46:25.778458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.778447    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d39646-f793-4e6c-a228-3bbe22942190" containerName="revoke-keys"
Nov 01 20:46:25.781836 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.781815    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.785700 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.785676    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:46:25.785873 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.785709    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:46:25.786054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.785815    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:46:25.788166 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.787953    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:46:25.788450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.788428    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"]
Nov 01 20:46:25.881603 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.881571    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.881773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.881685    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.881773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.881749    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvch\" (UniqueName: \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.982947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.982911    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvch\" (UniqueName: \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.983102 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.983037    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.983174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.983112    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.983817 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.983795    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.985428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.985410    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:25.992206 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:25.992179    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvch\" (UniqueName: \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\") pod \"maas-api-revoke-sub-3e31a506-rvnln\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:26.098029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:26.097995    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:26.374428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:26.374396    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"]
Nov 01 20:46:26.377736 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:46:26.377706    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f6212c_6d11_45b0_9289_5a8d48a0be67.slice/crio-85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838 WatchSource:0}: Error finding container 85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838: Status 404 returned error can't find the container with id 85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838
Nov 01 20:46:27.401093 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:27.401055    5041 generic.go:356] "Generic (PLEG): container finished" podID="02f6212c-6d11-45b0-9289-5a8d48a0be67" containerID="443614aaaa66a0a11b7da444ebea37d6d7ee75a34bcd5ce74ffbd5b13d3d92f6" exitCode=0
Nov 01 20:46:27.403194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:27.403164    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln" event={"ID":"02f6212c-6d11-45b0-9289-5a8d48a0be67","Type":"ContainerDied","Data":"443614aaaa66a0a11b7da444ebea37d6d7ee75a34bcd5ce74ffbd5b13d3d92f6"}
Nov 01 20:46:27.403447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:27.403410    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln" event={"ID":"02f6212c-6d11-45b0-9289-5a8d48a0be67","Type":"ContainerStarted","Data":"85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838"}
Nov 01 20:46:28.638512 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:46:28.638469    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:46:28.729860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.729834    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:28.829792 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.829753    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\") pod \"02f6212c-6d11-45b0-9289-5a8d48a0be67\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") "
Nov 01 20:46:28.829998 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.829878    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\") pod \"02f6212c-6d11-45b0-9289-5a8d48a0be67\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") "
Nov 01 20:46:28.830340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.830312    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca" pod "02f6212c-6d11-45b0-9289-5a8d48a0be67" (UID: "02f6212c-6d11-45b0-9289-5a8d48a0be67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:46:28.830815 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.830771    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:28.832692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.832659    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca" pod "02f6212c-6d11-45b0-9289-5a8d48a0be67" (UID: "02f6212c-6d11-45b0-9289-5a8d48a0be67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:28.931601 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.931557    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\" (UniqueName: \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\") pod \"02f6212c-6d11-45b0-9289-5a8d48a0be67\" (UID: \"02f6212c-6d11-45b0-9289-5a8d48a0be67\") "
Nov 01 20:46:28.931858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.931839    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f6212c-6d11-45b0-9289-5a8d48a0be67-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:28.933702 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:28.933679    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch" pod "02f6212c-6d11-45b0-9289-5a8d48a0be67" (UID: "02f6212c-6d11-45b0-9289-5a8d48a0be67"). InnerVolumeSpecName "kube-api-access-bzvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:29.032724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:29.032652    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvch\" (UniqueName: \"kubernetes.io/projected/02f6212c-6d11-45b0-9289-5a8d48a0be67-kube-api-access-bzvch\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:29.425008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:29.424978    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"
Nov 01 20:46:29.425187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:29.424977    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln" event={"ID":"02f6212c-6d11-45b0-9289-5a8d48a0be67","Type":"ContainerDied","Data":"85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838"}
Nov 01 20:46:29.425187 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:29.425105    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85071513f06b89562e22dccc1eb29aa79fe3bb37f296ffa35b2058211f5d9838"
Nov 01 20:46:37.433749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:37.433716    5041 scope.go:122] "RemoveContainer" containerID="741bb4d343c309d4f5b32478c8091ed0ab7845a5af4e7ce6b4ae00b0e53d9f46"
Nov 01 20:46:37.475054 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:37.475012    5041 scope.go:122] "RemoveContainer" containerID="3dcccc391f47ccb3190d5074a312081f48290fb453685c551f1309c7fceb93f3"
Nov 01 20:46:37.527722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:37.527694    5041 scope.go:122] "RemoveContainer" containerID="ebf9aeaaea5e655fd2b5f0eeccb3e0dce72823a52858d700416d4d56544046ff"
Nov 01 20:46:37.538366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:37.538335    5041 scope.go:122] "RemoveContainer" containerID="8a4853851b15a0bea0083a64339495a83862936b5cb1727a860ac94db8967e1a"
Nov 01 20:46:37.607810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:37.606970    5041 scope.go:122] "RemoveContainer" containerID="913fd5f71caf7b2201269472e5f2d8d78af3aff61483ea634b8397b4f330cd36"
Nov 01 20:46:43.531923 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.531888    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"]
Nov 01 20:46:43.532437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.532418    5041 cpu_manager.go:414] "containerMap: removing container" podUID="02f6212c-6d11-45b0-9289-5a8d48a0be67" containerName="revoke-keys"
Nov 01 20:46:43.532491 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.532441    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f6212c-6d11-45b0-9289-5a8d48a0be67" containerName="revoke-keys"
Nov 01 20:46:43.532631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.532617    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f6212c-6d11-45b0-9289-5a8d48a0be67" containerName="revoke-keys"
Nov 01 20:46:43.537711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.537690    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.541527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.541489    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:46:43.541991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.541967    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:46:43.542129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.542035    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:46:43.542129 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.542102    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:46:43.542400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.542377    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"]
Nov 01 20:46:43.637016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.636986    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.637207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.637021    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxv4\" (UniqueName: \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.637207 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.637060    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.738407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.738373    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.738407 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.738408    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxv4\" (UniqueName: \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.738676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.738433    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.739133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.739106    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.740885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.740864    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.745870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.745848    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxv4\" (UniqueName: \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\") pod \"maas-api-revoke-sub-5a19d8d0-s7fpp\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:43.855429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:43.855401    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:44.135831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:44.135808    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"]
Nov 01 20:46:44.137237 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:46:44.137208    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e356b18_7b63_44c9_bb3e_c6148053176a.slice/crio-c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61 WatchSource:0}: Error finding container c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61: Status 404 returned error can't find the container with id c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61
Nov 01 20:46:44.557448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:44.557413    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp" event={"ID":"0e356b18-7b63-44c9-bb3e-c6148053176a","Type":"ContainerStarted","Data":"884520a8ccbdb32c0af9642059266f9b25047f0e7d63003aca9690abc192a015"}
Nov 01 20:46:44.557861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:44.557468    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp" event={"ID":"0e356b18-7b63-44c9-bb3e-c6148053176a","Type":"ContainerStarted","Data":"c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61"}
Nov 01 20:46:44.574460 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:44.574407    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp" podStartSLOduration=1.574393885 podStartE2EDuration="1.574393885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:46:44.571644993 +0000 UTC m=+2892.292723671" watchObservedRunningTime="2026-11-01 20:46:44.574393885 +0000 UTC m=+2892.295472561"
Nov 01 20:46:45.564083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:45.564046    5041 generic.go:356] "Generic (PLEG): container finished" podID="0e356b18-7b63-44c9-bb3e-c6148053176a" containerID="884520a8ccbdb32c0af9642059266f9b25047f0e7d63003aca9690abc192a015" exitCode=0
Nov 01 20:46:45.564503 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:45.564137    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp" event={"ID":"0e356b18-7b63-44c9-bb3e-c6148053176a","Type":"ContainerDied","Data":"884520a8ccbdb32c0af9642059266f9b25047f0e7d63003aca9690abc192a015"}
Nov 01 20:46:46.718360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.718332    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:46.872831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.872732    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\" (UniqueName: \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\") pod \"0e356b18-7b63-44c9-bb3e-c6148053176a\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") "
Nov 01 20:46:46.873011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.872866    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\") pod \"0e356b18-7b63-44c9-bb3e-c6148053176a\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") "
Nov 01 20:46:46.873011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.872934    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\") pod \"0e356b18-7b63-44c9-bb3e-c6148053176a\" (UID: \"0e356b18-7b63-44c9-bb3e-c6148053176a\") "
Nov 01 20:46:46.873304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.873273    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca" pod "0e356b18-7b63-44c9-bb3e-c6148053176a" (UID: "0e356b18-7b63-44c9-bb3e-c6148053176a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:46:46.874932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.874910    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4" pod "0e356b18-7b63-44c9-bb3e-c6148053176a" (UID: "0e356b18-7b63-44c9-bb3e-c6148053176a"). InnerVolumeSpecName "kube-api-access-fpxv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:46:46.875029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.874988    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca" pod "0e356b18-7b63-44c9-bb3e-c6148053176a" (UID: "0e356b18-7b63-44c9-bb3e-c6148053176a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:46:46.974795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.974743    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:46.974979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.974828    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxv4\" (UniqueName: \"kubernetes.io/projected/0e356b18-7b63-44c9-bb3e-c6148053176a-kube-api-access-fpxv4\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:46.974979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:46.974845    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e356b18-7b63-44c9-bb3e-c6148053176a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:46:47.580822 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:47.580794    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"
Nov 01 20:46:47.581024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:47.580765    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp" event={"ID":"0e356b18-7b63-44c9-bb3e-c6148053176a","Type":"ContainerDied","Data":"c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61"}
Nov 01 20:46:47.581024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:46:47.580936    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7450afde5872448a9e949ca5d43e7b93afe6187f17e70f7889bf0e169b36c61"
Nov 01 20:46:58.541074 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:46:58.541038    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:47:01.020338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:01.020302    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"]
Nov 01 20:47:01.022749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:01.022726    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-sfk74"]
Nov 01 20:47:02.429523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:02.429483    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2ffa87-b809-41d2-bea3-6e870f3e9969" path="/var/lib/kubelet/pods/6c2ffa87-b809-41d2-bea3-6e870f3e9969/volumes"
Nov 01 20:47:14.985045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.984966    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:47:14.985434 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.985427    5041 cpu_manager.go:414] "containerMap: removing container" podUID="0e356b18-7b63-44c9-bb3e-c6148053176a" containerName="revoke-keys"
Nov 01 20:47:14.985477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.985438    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e356b18-7b63-44c9-bb3e-c6148053176a" containerName="revoke-keys"
Nov 01 20:47:14.985548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.985535    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e356b18-7b63-44c9-bb3e-c6148053176a" containerName="revoke-keys"
Nov 01 20:47:14.989992 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.989974    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:14.995435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.995407    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:47:14.997749 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.997727    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c268d1\""
Nov 01 20:47:14.998224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:14.997811    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c268d1-dockercfg-n4nqv\""
Nov 01 20:47:15.026289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.026257    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:47:15.029971 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.029950    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.042844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.042814    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48t88\" (UniqueName: \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.042980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.042919    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.042980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.042971    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scv2k\" (UniqueName: \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.043112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.043002    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.046008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.045981    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:47:15.144089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.144054    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48t88\" (UniqueName: \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.144313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.144156    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.144313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.144185    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scv2k\" (UniqueName: \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.144313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.144212    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.145025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.145000    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.145025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.145017    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.151651 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.151619    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48t88\" (UniqueName: \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\") pod \"payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.152013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.151988    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scv2k\" (UniqueName: \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\") pod \"payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.305505 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.305471    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:15.350172 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.350138    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:15.587199 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.587173    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:47:15.589125 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:47:15.589102    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf7b71f_e8fa_4174_b80d_3b362bf73047.slice/crio-5e4b3f2564e451a2647f52072938ccae0456a4e6070d237d4d8155c9c9c7bb2b WatchSource:0}: Error finding container 5e4b3f2564e451a2647f52072938ccae0456a4e6070d237d4d8155c9c9c7bb2b: Status 404 returned error can't find the container with id 5e4b3f2564e451a2647f52072938ccae0456a4e6070d237d4d8155c9c9c7bb2b
Nov 01 20:47:15.646865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.646836    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:47:15.648001 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:47:15.647978    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd63c64e_5381_4373_9995_466f3680a021.slice/crio-55a7c5928768b2afafbc462c66382f007c31705e2646c8ef7010a2233f65fbd9 WatchSource:0}: Error finding container 55a7c5928768b2afafbc462c66382f007c31705e2646c8ef7010a2233f65fbd9: Status 404 returned error can't find the container with id 55a7c5928768b2afafbc462c66382f007c31705e2646c8ef7010a2233f65fbd9
Nov 01 20:47:15.747084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.747042    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" event={"ID":"fd63c64e-5381-4373-9995-466f3680a021","Type":"ContainerStarted","Data":"55a7c5928768b2afafbc462c66382f007c31705e2646c8ef7010a2233f65fbd9"}
Nov 01 20:47:15.748348 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:15.748316    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" event={"ID":"ebf7b71f-e8fa-4174-b80d-3b362bf73047","Type":"ContainerStarted","Data":"5e4b3f2564e451a2647f52072938ccae0456a4e6070d237d4d8155c9c9c7bb2b"}
Nov 01 20:47:17.767400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.767341    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" event={"ID":"fd63c64e-5381-4373-9995-466f3680a021","Type":"ContainerStarted","Data":"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"}
Nov 01 20:47:17.768121 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.768098    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:17.770819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.770151    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" event={"ID":"ebf7b71f-e8fa-4174-b80d-3b362bf73047","Type":"ContainerStarted","Data":"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"}
Nov 01 20:47:17.770819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.770212    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:17.784448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.784386    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" podStartSLOduration=0.786036494 podStartE2EDuration="2.784368639s" totalImagesPullingTime="1.998332145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:17.782013406 +0000 UTC m=+2925.503092083" watchObservedRunningTime="2026-11-01 20:47:17.784368639 +0000 UTC m=+2925.505447317"
Nov 01 20:47:17.797025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:17.796976    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" podStartSLOduration=1.786333132 podStartE2EDuration="3.796962685s" totalImagesPullingTime="2.010629553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:17.796643399 +0000 UTC m=+2925.517722076" watchObservedRunningTime="2026-11-01 20:47:17.796962685 +0000 UTC m=+2925.518041361"
Nov 01 20:47:23.022825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:23.022748    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"]
Nov 01 20:47:23.027143 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:23.027111    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cj5kg"]
Nov 01 20:47:24.427438 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:24.427406    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35de46f-4fc4-443c-990f-75e1e0d1d843" path="/var/lib/kubelet/pods/b35de46f-4fc4-443c-990f-75e1e0d1d843/volumes"
Nov 01 20:47:28.541886 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:47:28.541849    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:47:28.779714 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:28.779685    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:47:29.785881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:29.785851    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:47:37.670459 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:37.670424    5041 scope.go:122] "RemoveContainer" containerID="a736ca29159166b25e8890080813333e31b515c9b667b91d2f71c9c0b3b69df2"
Nov 01 20:47:37.680286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:37.680264    5041 scope.go:122] "RemoveContainer" containerID="53196fa755567c9a569cb455c3f75f39ee64a768db0582d9a2f52b649020ba23"
Nov 01 20:47:39.022652 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:39.022615    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"]
Nov 01 20:47:39.025877 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:39.025855    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-slc92"]
Nov 01 20:47:40.427067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:40.427033    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0430059a-6cbb-4309-9aee-49aaa49691ed" path="/var/lib/kubelet/pods/0430059a-6cbb-4309-9aee-49aaa49691ed/volumes"
Nov 01 20:47:49.019391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:49.019351    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"]
Nov 01 20:47:49.021697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:49.021662    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gbz92"]
Nov 01 20:47:50.427156 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:50.427122    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f59b4e-4981-4cd2-acf7-719fd8d105af" path="/var/lib/kubelet/pods/55f59b4e-4981-4cd2-acf7-719fd8d105af/volumes"
Nov 01 20:47:55.700490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.700456    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:47:55.706962 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.706942    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.711066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.711044    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:47:55.711066 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.711059    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a97e34\""
Nov 01 20:47:55.712159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.712134    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a97e34-dockercfg-2gw4v\""
Nov 01 20:47:55.728443 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.728416    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:47:55.732809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.732789    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.743304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.743281    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:47:55.756473 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.756447    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7ck\" (UniqueName: \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.756625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.756478    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.756625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.756580    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.756730 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.756664    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2hg\" (UniqueName: \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.858024 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.857987    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.858235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.858065    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.858235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.858085    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2hg\" (UniqueName: \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.858235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.858203    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7ck\" (UniqueName: \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.858746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.858723    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.858865 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.858806    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:55.866260 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.866228    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7ck\" (UniqueName: \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:55.866394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:55.866273    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2hg\" (UniqueName: \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\") pod \"payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:56.023111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:56.023026    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:56.050628 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:56.050589    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:56.307722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:56.303840    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:47:56.344010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:56.343987    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:47:56.355184 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:47:56.355150    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd58ea31_2a0b_4baf_8be2_14d87d895ec8.slice/crio-57ecb1526aace46b052bf325004d94e4b1069084cdd0d677a20d1954766d6fa6 WatchSource:0}: Error finding container 57ecb1526aace46b052bf325004d94e4b1069084cdd0d677a20d1954766d6fa6: Status 404 returned error can't find the container with id 57ecb1526aace46b052bf325004d94e4b1069084cdd0d677a20d1954766d6fa6
Nov 01 20:47:57.021444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.021404    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" event={"ID":"dd58ea31-2a0b-4baf-8be2-14d87d895ec8","Type":"ContainerStarted","Data":"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"}
Nov 01 20:47:57.021948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.021460    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" event={"ID":"dd58ea31-2a0b-4baf-8be2-14d87d895ec8","Type":"ContainerStarted","Data":"57ecb1526aace46b052bf325004d94e4b1069084cdd0d677a20d1954766d6fa6"}
Nov 01 20:47:57.021948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.021492    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:47:57.023163 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.023138    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" event={"ID":"31183fb0-f235-4ccc-bd8e-f145e0172967","Type":"ContainerStarted","Data":"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"}
Nov 01 20:47:57.023286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.023182    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" event={"ID":"31183fb0-f235-4ccc-bd8e-f145e0172967","Type":"ContainerStarted","Data":"26ef841aa37669729fbafc89c61c355572c5e0b34aed1fc96a0cb4f53b03cc5a"}
Nov 01 20:47:57.023286 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.023230    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:47:57.040324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.040273    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" podStartSLOduration=2.040257241 podStartE2EDuration="2.040257241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:57.036214573 +0000 UTC m=+2964.757293251" watchObservedRunningTime="2026-11-01 20:47:57.040257241 +0000 UTC m=+2964.761335918"
Nov 01 20:47:57.051936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:47:57.051895    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" podStartSLOduration=2.051883537 podStartE2EDuration="2.051883537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:57.050333366 +0000 UTC m=+2964.771412040" watchObservedRunningTime="2026-11-01 20:47:57.051883537 +0000 UTC m=+2964.772962212"
Nov 01 20:47:58.541185 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:47:58.541152    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:48:08.031165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:08.031136    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:48:08.031611 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:08.031213    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:48:15.294636 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:15.294605    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r5wf2_ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f/payload-processing/0.log"
Nov 01 20:48:15.458621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:15.458592    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r5wf2_ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f/payload-processing/0.log"
Nov 01 20:48:15.678365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:15.678279    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr_dd58ea31-2a0b-4baf-8be2-14d87d895ec8/payload-processing/0.log"
Nov 01 20:48:28.542139 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:48:28.542101    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:48:33.084980 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:33.084953    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:48:33.098858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:33.098833    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:48:37.712574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:37.712535    5041 scope.go:122] "RemoveContainer" containerID="c06b97fe8076085c1bfaa08f79c967980e73eb5a4baf73f9d634bf746f8320ea"
Nov 01 20:48:37.724537 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:37.724513    5041 scope.go:122] "RemoveContainer" containerID="947e334cde5aad53fd8a41046fed48e7622274ac8b007ea780fe4a1af3846b36"
Nov 01 20:48:37.733256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:37.733236    5041 scope.go:122] "RemoveContainer" containerID="e7a9a61fd2f2d3005c4af633c9c4f36043e316a8e6d4b255de683b4a48820578"
Nov 01 20:48:50.667632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:50.667549    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c_fd63c64e-5381-4373-9995-466f3680a021/payload-processing/0.log"
Nov 01 20:48:50.845067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:50.845027    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r5wf2_ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f/payload-processing/0.log"
Nov 01 20:48:51.060167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:51.060139    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-r5wf2_ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f/payload-processing/0.log"
Nov 01 20:48:51.232847 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:48:51.232817    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr_dd58ea31-2a0b-4baf-8be2-14d87d895ec8/payload-processing/0.log"
Nov 01 20:48:58.541764 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:48:58.541727    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:49:05.023608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:05.023568    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"]
Nov 01 20:49:05.026011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:05.025988    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4db39d7-fw59q"]
Nov 01 20:49:06.429231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:06.429191    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d29371-06bf-49b8-8355-342b47b71acf" path="/var/lib/kubelet/pods/10d29371-06bf-49b8-8355-342b47b71acf/volumes"
Nov 01 20:49:28.548847 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:49:28.548803    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:49:33.719311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.719275    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"]
Nov 01 20:49:33.723027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.723009    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.727242 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.727213    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:49:33.727393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.727281    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:49:33.727457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.727403    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:49:33.727527 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.727512    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:49:33.729340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.729314    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"]
Nov 01 20:49:33.885996 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.885955    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.886184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.886083    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xs9n\" (UniqueName: \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.886184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.886164    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.987514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.987431    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xs9n\" (UniqueName: \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.987679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.987519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.987679 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.987590    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.988339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.988316    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.990136 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.990110    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:33.995325 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:33.995298    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xs9n\" (UniqueName: \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\") pod \"maas-api-revoke-sub-a7122df3-mpltg\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:34.039046 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:34.039013    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:34.322675 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:34.322646    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"]
Nov 01 20:49:34.324716 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:49:34.324685    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42638d25_38c5_463a_90f0_e2ff67bf9886.slice/crio-b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe WatchSource:0}: Error finding container b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe: Status 404 returned error can't find the container with id b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe
Nov 01 20:49:34.614793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:34.614747    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg" event={"ID":"42638d25-38c5-463a-90f0-e2ff67bf9886","Type":"ContainerStarted","Data":"b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe"}
Nov 01 20:49:35.621964 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:35.621929    5041 generic.go:356] "Generic (PLEG): container finished" podID="42638d25-38c5-463a-90f0-e2ff67bf9886" containerID="2749cf1251cab2f21ce94803d4287142aba23e7f134a4f161703f6932d00fff5" exitCode=0
Nov 01 20:49:35.622360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:35.621981    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg" event={"ID":"42638d25-38c5-463a-90f0-e2ff67bf9886","Type":"ContainerDied","Data":"2749cf1251cab2f21ce94803d4287142aba23e7f134a4f161703f6932d00fff5"}
Nov 01 20:49:36.802240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.802215    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:36.919204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.919115    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\") pod \"42638d25-38c5-463a-90f0-e2ff67bf9886\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") "
Nov 01 20:49:36.919370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.919214    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\" (UniqueName: \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\") pod \"42638d25-38c5-463a-90f0-e2ff67bf9886\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") "
Nov 01 20:49:36.919370 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.919339    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\") pod \"42638d25-38c5-463a-90f0-e2ff67bf9886\" (UID: \"42638d25-38c5-463a-90f0-e2ff67bf9886\") "
Nov 01 20:49:36.919721 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.919694    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca" pod "42638d25-38c5-463a-90f0-e2ff67bf9886" (UID: "42638d25-38c5-463a-90f0-e2ff67bf9886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:49:36.921513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.921485    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n" pod "42638d25-38c5-463a-90f0-e2ff67bf9886" (UID: "42638d25-38c5-463a-90f0-e2ff67bf9886"). InnerVolumeSpecName "kube-api-access-8xs9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:36.921763 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:36.921729    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca" pod "42638d25-38c5-463a-90f0-e2ff67bf9886" (UID: "42638d25-38c5-463a-90f0-e2ff67bf9886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:49:37.020567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.020533    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:37.020567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.020564    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42638d25-38c5-463a-90f0-e2ff67bf9886-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:37.020567 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.020573    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xs9n\" (UniqueName: \"kubernetes.io/projected/42638d25-38c5-463a-90f0-e2ff67bf9886-kube-api-access-8xs9n\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:37.636806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.636765    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"
Nov 01 20:49:37.637020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.636814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg" event={"ID":"42638d25-38c5-463a-90f0-e2ff67bf9886","Type":"ContainerDied","Data":"b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe"}
Nov 01 20:49:37.637020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.636866    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4bc840f0f91e0187ae1ce081eed8e59276e5796c819f150aed2056f1a9eb7fe"
Nov 01 20:49:37.765241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:37.765207    5041 scope.go:122] "RemoveContainer" containerID="ce2921a80b653f8ce1421d4d3b4630f7151d1bb5dcc8cf129e57406a7834ab48"
Nov 01 20:49:49.360223 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.360188    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:49:49.360863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.360438    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" podUID="fd63c64e-5381-4373-9995-466f3680a021" containerName="payload-processing" containerID="cri-o://f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f" gracePeriod=30
Nov 01 20:49:49.367318 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.367287    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:49:49.367518 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.367486    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" podUID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" containerName="payload-pre-processing" containerID="cri-o://36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb" gracePeriod=30
Nov 01 20:49:49.657575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.657549    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:49:49.683074 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.683048    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:49:49.709312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.709280    5041 generic.go:356] "Generic (PLEG): container finished" podID="fd63c64e-5381-4373-9995-466f3680a021" containerID="f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f" exitCode=0
Nov 01 20:49:49.709481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.709330    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"
Nov 01 20:49:49.709481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.709364    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" event={"ID":"fd63c64e-5381-4373-9995-466f3680a021","Type":"ContainerDied","Data":"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"}
Nov 01 20:49:49.709481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.709424    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c" event={"ID":"fd63c64e-5381-4373-9995-466f3680a021","Type":"ContainerDied","Data":"55a7c5928768b2afafbc462c66382f007c31705e2646c8ef7010a2233f65fbd9"}
Nov 01 20:49:49.709481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.709453    5041 scope.go:122] "RemoveContainer" containerID="f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"
Nov 01 20:49:49.711235 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.711213    5041 generic.go:356] "Generic (PLEG): container finished" podID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" containerID="36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb" exitCode=0
Nov 01 20:49:49.711351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.711261    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"
Nov 01 20:49:49.711351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.711278    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" event={"ID":"ebf7b71f-e8fa-4174-b80d-3b362bf73047","Type":"ContainerDied","Data":"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"}
Nov 01 20:49:49.711351 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.711321    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h" event={"ID":"ebf7b71f-e8fa-4174-b80d-3b362bf73047","Type":"ContainerDied","Data":"5e4b3f2564e451a2647f52072938ccae0456a4e6070d237d4d8155c9c9c7bb2b"}
Nov 01 20:49:49.719578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.719558    5041 scope.go:122] "RemoveContainer" containerID="f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"
Nov 01 20:49:49.719900 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:49:49.719874    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f\": container with ID starting with f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f not found: ID does not exist" containerID="f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"
Nov 01 20:49:49.719993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.719903    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f"} err="failed to get container status \"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f\": rpc error: code = NotFound desc = could not find container \"f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f\": container with ID starting with f65ace3bbee041262e4f54fdfde90d4faf3b4f89a7042d22db38b5e188d3671f not found: ID does not exist"
Nov 01 20:49:49.719993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.719920    5041 scope.go:122] "RemoveContainer" containerID="36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"
Nov 01 20:49:49.733004 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.732979    5041 scope.go:122] "RemoveContainer" containerID="36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"
Nov 01 20:49:49.733445 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:49:49.733410    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb\": container with ID starting with 36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb not found: ID does not exist" containerID="36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"
Nov 01 20:49:49.733587 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.733456    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb"} err="failed to get container status \"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb\": rpc error: code = NotFound desc = could not find container \"36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb\": container with ID starting with 36cee3470a99af6bef8c3201caea56150c2840b88bfaa4ff0dfd326042b560bb not found: ID does not exist"
Nov 01 20:49:49.855883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.855851    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\" (UniqueName: \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\") pod \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") "
Nov 01 20:49:49.856100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.855991    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\") pod \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\" (UID: \"ebf7b71f-e8fa-4174-b80d-3b362bf73047\") "
Nov 01 20:49:49.856100 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.856085    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\" (UniqueName: \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\") pod \"fd63c64e-5381-4373-9995-466f3680a021\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") "
Nov 01 20:49:49.856230 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.856177    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\") pod \"fd63c64e-5381-4373-9995-466f3680a021\" (UID: \"fd63c64e-5381-4373-9995-466f3680a021\") "
Nov 01 20:49:49.856465 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.856439    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume" pod "ebf7b71f-e8fa-4174-b80d-3b362bf73047" (UID: "ebf7b71f-e8fa-4174-b80d-3b362bf73047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:49:49.856622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.856598    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume" pod "fd63c64e-5381-4373-9995-466f3680a021" (UID: "fd63c64e-5381-4373-9995-466f3680a021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:49:49.858076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.858053    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k" pod "ebf7b71f-e8fa-4174-b80d-3b362bf73047" (UID: "ebf7b71f-e8fa-4174-b80d-3b362bf73047"). InnerVolumeSpecName "kube-api-access-scv2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:49.858183 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.858166    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88" pod "fd63c64e-5381-4373-9995-466f3680a021" (UID: "fd63c64e-5381-4373-9995-466f3680a021"). InnerVolumeSpecName "kube-api-access-48t88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:49:49.957859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.957747    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf7b71f-e8fa-4174-b80d-3b362bf73047-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.957859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.957789    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48t88\" (UniqueName: \"kubernetes.io/projected/fd63c64e-5381-4373-9995-466f3680a021-kube-api-access-48t88\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.957859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.957799    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd63c64e-5381-4373-9995-466f3680a021-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:49.957859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:49.957808    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scv2k\" (UniqueName: \"kubernetes.io/projected/ebf7b71f-e8fa-4174-b80d-3b362bf73047-kube-api-access-scv2k\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:49:50.032635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.032605    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:49:50.035016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.034991    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c268d1-dd475596b-z9s9c"]
Nov 01 20:49:50.043538 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.043514    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:49:50.047091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.047063    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c268d1-76998665f-f725h"]
Nov 01 20:49:50.427710 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.427674    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" path="/var/lib/kubelet/pods/ebf7b71f-e8fa-4174-b80d-3b362bf73047/volumes"
Nov 01 20:49:50.428313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:49:50.428243    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd63c64e-5381-4373-9995-466f3680a021" path="/var/lib/kubelet/pods/fd63c64e-5381-4373-9995-466f3680a021/volumes"
Nov 01 20:49:58.540877 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:49:58.540841    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:50:04.729963 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:04.729923    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:50:04.730496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:04.730195    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" podUID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" containerName="payload-processing" containerID="cri-o://18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398" gracePeriod=30
Nov 01 20:50:04.740990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:04.740960    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:50:04.741205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:04.741177    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" podUID="31183fb0-f235-4ccc-bd8e-f145e0172967" containerName="payload-pre-processing" containerID="cri-o://94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0" gracePeriod=30
Nov 01 20:50:05.028881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.028855    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:50:05.065394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.065360    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:50:05.111122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111089    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\" (UniqueName: \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\") pod \"31183fb0-f235-4ccc-bd8e-f145e0172967\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") "
Nov 01 20:50:05.111312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111130    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\" (UniqueName: \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\") pod \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") "
Nov 01 20:50:05.111312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111275    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\") pod \"31183fb0-f235-4ccc-bd8e-f145e0172967\" (UID: \"31183fb0-f235-4ccc-bd8e-f145e0172967\") "
Nov 01 20:50:05.111312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111314    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\") pod \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\" (UID: \"dd58ea31-2a0b-4baf-8be2-14d87d895ec8\") "
Nov 01 20:50:05.111724 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111695    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume" pod "31183fb0-f235-4ccc-bd8e-f145e0172967" (UID: "31183fb0-f235-4ccc-bd8e-f145e0172967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:05.111905 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.111876    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume" pod "dd58ea31-2a0b-4baf-8be2-14d87d895ec8" (UID: "dd58ea31-2a0b-4baf-8be2-14d87d895ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:05.113889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.113861    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg" pod "dd58ea31-2a0b-4baf-8be2-14d87d895ec8" (UID: "dd58ea31-2a0b-4baf-8be2-14d87d895ec8"). InnerVolumeSpecName "kube-api-access-2f2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:05.113889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.113870    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck" pod "31183fb0-f235-4ccc-bd8e-f145e0172967" (UID: "31183fb0-f235-4ccc-bd8e-f145e0172967"). InnerVolumeSpecName "kube-api-access-9g7ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:05.212918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.212885    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31183fb0-f235-4ccc-bd8e-f145e0172967-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.212918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.212911    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.212918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.212923    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7ck\" (UniqueName: \"kubernetes.io/projected/31183fb0-f235-4ccc-bd8e-f145e0172967-kube-api-access-9g7ck\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.213167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.212934    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2hg\" (UniqueName: \"kubernetes.io/projected/dd58ea31-2a0b-4baf-8be2-14d87d895ec8-kube-api-access-2f2hg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:05.811446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.811410    5041 generic.go:356] "Generic (PLEG): container finished" podID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" containerID="18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398" exitCode=0
Nov 01 20:50:05.811921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.811455    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"
Nov 01 20:50:05.811921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.811506    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" event={"ID":"dd58ea31-2a0b-4baf-8be2-14d87d895ec8","Type":"ContainerDied","Data":"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"}
Nov 01 20:50:05.811921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.811554    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr" event={"ID":"dd58ea31-2a0b-4baf-8be2-14d87d895ec8","Type":"ContainerDied","Data":"57ecb1526aace46b052bf325004d94e4b1069084cdd0d677a20d1954766d6fa6"}
Nov 01 20:50:05.811921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.811576    5041 scope.go:122] "RemoveContainer" containerID="18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"
Nov 01 20:50:05.813071 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.813049    5041 generic.go:356] "Generic (PLEG): container finished" podID="31183fb0-f235-4ccc-bd8e-f145e0172967" containerID="94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0" exitCode=0
Nov 01 20:50:05.813144 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.813104    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"
Nov 01 20:50:05.813194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.813131    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" event={"ID":"31183fb0-f235-4ccc-bd8e-f145e0172967","Type":"ContainerDied","Data":"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"}
Nov 01 20:50:05.813194 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.813186    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq" event={"ID":"31183fb0-f235-4ccc-bd8e-f145e0172967","Type":"ContainerDied","Data":"26ef841aa37669729fbafc89c61c355572c5e0b34aed1fc96a0cb4f53b03cc5a"}
Nov 01 20:50:05.823161 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.823140    5041 scope.go:122] "RemoveContainer" containerID="18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"
Nov 01 20:50:05.823568 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:05.823442    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398\": container with ID starting with 18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398 not found: ID does not exist" containerID="18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"
Nov 01 20:50:05.823568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.823469    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398"} err="failed to get container status \"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398\": rpc error: code = NotFound desc = could not find container \"18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398\": container with ID starting with 18dd847c40964d2a7d8f52a0a4531d18a12cb81470531b81560559018ef2e398 not found: ID does not exist"
Nov 01 20:50:05.823568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.823486    5041 scope.go:122] "RemoveContainer" containerID="94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"
Nov 01 20:50:05.833544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.833520    5041 scope.go:122] "RemoveContainer" containerID="94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"
Nov 01 20:50:05.833822 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:05.833801    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0\": container with ID starting with 94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0 not found: ID does not exist" containerID="94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"
Nov 01 20:50:05.833898 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.833827    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0"} err="failed to get container status \"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0\": rpc error: code = NotFound desc = could not find container \"94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0\": container with ID starting with 94ba5a946a34971912f3cef084e432477423567510f97355be79146a2aafc4b0 not found: ID does not exist"
Nov 01 20:50:05.834908 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.834889    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:50:05.838335 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.838314    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-b6b9dc6b6-6rptr"]
Nov 01 20:50:05.850916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.850274    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:50:05.856096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:05.856068    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-758556cc6c-f77mq"]
Nov 01 20:50:06.428050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:06.428018    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31183fb0-f235-4ccc-bd8e-f145e0172967" path="/var/lib/kubelet/pods/31183fb0-f235-4ccc-bd8e-f145e0172967/volumes"
Nov 01 20:50:06.428398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:06.428385    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" path="/var/lib/kubelet/pods/dd58ea31-2a0b-4baf-8be2-14d87d895ec8/volumes"
Nov 01 20:50:09.020932 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:09.020896    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"]
Nov 01 20:50:09.023205 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:09.023179    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48d306b-pzbxl"]
Nov 01 20:50:10.426768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:10.426733    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f41ab52-7a07-488a-93aa-1482fba8c794" path="/var/lib/kubelet/pods/0f41ab52-7a07-488a-93aa-1482fba8c794/volumes"
Nov 01 20:50:24.153180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:24.153097    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:50:28.541145 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:28.541104    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:50:29.154067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.154013    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" podUID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" containerName="payload-pre-processing" containerID="cri-o://187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0" gracePeriod=25
Nov 01 20:50:29.406697 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.406641    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:50:29.470582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.470545    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\" (UniqueName: \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\") pod \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") "
Nov 01 20:50:29.470803 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.470710    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\") pod \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\" (UID: \"49da114b-50b0-4852-a9a8-8c36be0ac8d4\") "
Nov 01 20:50:29.471096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.471068    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume" pod "49da114b-50b0-4852-a9a8-8c36be0ac8d4" (UID: "49da114b-50b0-4852-a9a8-8c36be0ac8d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:29.472622 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.472601    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d" pod "49da114b-50b0-4852-a9a8-8c36be0ac8d4" (UID: "49da114b-50b0-4852-a9a8-8c36be0ac8d4"). InnerVolumeSpecName "kube-api-access-prf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:29.571581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.571546    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49da114b-50b0-4852-a9a8-8c36be0ac8d4-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:29.571581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.571575    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prf9d\" (UniqueName: \"kubernetes.io/projected/49da114b-50b0-4852-a9a8-8c36be0ac8d4-kube-api-access-prf9d\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:29.954477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.954440    5041 generic.go:356] "Generic (PLEG): container finished" podID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" containerID="187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0" exitCode=0
Nov 01 20:50:29.954664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.954486    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"
Nov 01 20:50:29.954664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.954508    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" event={"ID":"49da114b-50b0-4852-a9a8-8c36be0ac8d4","Type":"ContainerDied","Data":"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"}
Nov 01 20:50:29.954664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.954552    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh" event={"ID":"49da114b-50b0-4852-a9a8-8c36be0ac8d4","Type":"ContainerDied","Data":"6954c5f688faaf3bcab46e884c9cb08ceb3d6448e4db69585e51892012ec5c3b"}
Nov 01 20:50:29.954664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.954573    5041 scope.go:122] "RemoveContainer" containerID="187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"
Nov 01 20:50:29.964105 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.964087    5041 scope.go:122] "RemoveContainer" containerID="187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"
Nov 01 20:50:29.964332 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:29.964314    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0\": container with ID starting with 187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0 not found: ID does not exist" containerID="187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"
Nov 01 20:50:29.964377 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.964340    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0"} err="failed to get container status \"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0\": rpc error: code = NotFound desc = could not find container \"187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0\": container with ID starting with 187aef2613a76c31f4479d34cf0ba0a1246a7d093e85a8c0f552768adcae72c0 not found: ID does not exist"
Nov 01 20:50:29.975926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.975898    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:50:29.979682 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:29.979658    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wwqfh"]
Nov 01 20:50:30.428189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:30.428158    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" path="/var/lib/kubelet/pods/49da114b-50b0-4852-a9a8-8c36be0ac8d4/volumes"
Nov 01 20:50:37.788727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:37.788688    5041 scope.go:122] "RemoveContainer" containerID="eadcc1f328db762f7f919f3a69c4576dadd9fdcf549df2187af71f86bbfc2592"
Nov 01 20:50:38.220902 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:38.220813    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:50:43.221719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.221654    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" podUID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" containerName="payload-processing" containerID="cri-o://a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea" gracePeriod=25
Nov 01 20:50:43.486266 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.486203    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:50:43.610147 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.610094    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\" (UniqueName: \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\") pod \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") "
Nov 01 20:50:43.610437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.610212    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\") pod \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\" (UID: \"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f\") "
Nov 01 20:50:43.610956 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.610929    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume" pod "ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" (UID: "ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:50:43.612934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.612904    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc" pod "ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" (UID: "ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f"). InnerVolumeSpecName "kube-api-access-ghzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:50:43.712096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.712064    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzgc\" (UniqueName: \"kubernetes.io/projected/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-kube-api-access-ghzgc\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:43.712096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:43.712091    5041 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f-plugins-config-volume\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:50:44.037451 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.037416    5041 generic.go:356] "Generic (PLEG): container finished" podID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" containerID="a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea" exitCode=0
Nov 01 20:50:44.037664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.037457    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"
Nov 01 20:50:44.037664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.037471    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" event={"ID":"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f","Type":"ContainerDied","Data":"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"}
Nov 01 20:50:44.037664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.037513    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-r5wf2" event={"ID":"ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f","Type":"ContainerDied","Data":"daf50433a6aee26319b81ee10dd27fd189cea14a31a599c3d23312920d0d6738"}
Nov 01 20:50:44.037664 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.037536    5041 scope.go:122] "RemoveContainer" containerID="a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"
Nov 01 20:50:44.049719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.049687    5041 scope.go:122] "RemoveContainer" containerID="a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"
Nov 01 20:50:44.050173 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:44.050147    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea\": container with ID starting with a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea not found: ID does not exist" containerID="a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"
Nov 01 20:50:44.050289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.050184    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea"} err="failed to get container status \"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea\": rpc error: code = NotFound desc = could not find container \"a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea\": container with ID starting with a2393b96d82b1d3dd26d86eb8e5bc6dd55187a8db28708da32bb1b8623b7ceea not found: ID does not exist"
Nov 01 20:50:44.064184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.064153    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:50:44.067480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.067457    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-r5wf2"]
Nov 01 20:50:44.426683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:50:44.426606    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" path="/var/lib/kubelet/pods/ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f/volumes"
Nov 01 20:50:58.548228 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:50:58.548171    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:51:14.022067 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:14.022033    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"]
Nov 01 20:51:14.025573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:14.025540    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6fvxp"]
Nov 01 20:51:14.427276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:14.427242    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d39646-f793-4e6c-a228-3bbe22942190" path="/var/lib/kubelet/pods/72d39646-f793-4e6c-a228-3bbe22942190/volumes"
Nov 01 20:51:28.540862 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:51:28.540821    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:51:29.021941 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:29.021842    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"]
Nov 01 20:51:29.024850 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:29.024818    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rvnln"]
Nov 01 20:51:30.427333 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:30.427303    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f6212c-6d11-45b0-9289-5a8d48a0be67" path="/var/lib/kubelet/pods/02f6212c-6d11-45b0-9289-5a8d48a0be67/volumes"
Nov 01 20:51:37.830490 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:37.830457    5041 scope.go:122] "RemoveContainer" containerID="1257dc7f7c4947db392df389ea8e98c675fb069b905d34d3ee939d7d7cb2ea26"
Nov 01 20:51:47.023320 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:47.023229    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"]
Nov 01 20:51:47.025296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:47.025271    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a19d8d0-s7fpp"]
Nov 01 20:51:48.434383 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:51:48.434343    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e356b18-7b63-44c9-bb3e-c6148053176a" path="/var/lib/kubelet/pods/0e356b18-7b63-44c9-bb3e-c6148053176a/volumes"
Nov 01 20:51:58.541141 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:51:58.541099    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:52:17.788229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788196    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"]
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788617    5041 cpu_manager.go:414] "containerMap: removing container" podUID="42638d25-38c5-463a-90f0-e2ff67bf9886" containerName="revoke-keys"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788627    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42638d25-38c5-463a-90f0-e2ff67bf9886" containerName="revoke-keys"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788648    5041 cpu_manager.go:414] "containerMap: removing container" podUID="fd63c64e-5381-4373-9995-466f3680a021" containerName="payload-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788655    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd63c64e-5381-4373-9995-466f3680a021" containerName="payload-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788662    5041 cpu_manager.go:414] "containerMap: removing container" podUID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" containerName="payload-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788668    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" containerName="payload-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788685    5041 cpu_manager.go:414] "containerMap: removing container" podUID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" containerName="payload-pre-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788691    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" containerName="payload-pre-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788699    5041 cpu_manager.go:414] "containerMap: removing container" podUID="31183fb0-f235-4ccc-bd8e-f145e0172967" containerName="payload-pre-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788703    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31183fb0-f235-4ccc-bd8e-f145e0172967" containerName="payload-pre-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788718    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" containerName="payload-processing"
Nov 01 20:52:17.788718 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788722    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" containerName="payload-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788732    5041 cpu_manager.go:414] "containerMap: removing container" podUID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" containerName="payload-pre-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788737    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" containerName="payload-pre-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788824    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="49da114b-50b0-4852-a9a8-8c36be0ac8d4" containerName="payload-pre-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788833    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="31183fb0-f235-4ccc-bd8e-f145e0172967" containerName="payload-pre-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788841    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3a3b40-cb74-4cbc-824f-d1dc7bacf92f" containerName="payload-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788848    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf7b71f-e8fa-4174-b80d-3b362bf73047" containerName="payload-pre-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788854    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="42638d25-38c5-463a-90f0-e2ff67bf9886" containerName="revoke-keys"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788862    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd63c64e-5381-4373-9995-466f3680a021" containerName="payload-processing"
Nov 01 20:52:17.789134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.788866    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd58ea31-2a0b-4baf-8be2-14d87d895ec8" containerName="payload-processing"
Nov 01 20:52:17.791978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.791961    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.796892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.796865    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:52:17.797035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.796868    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:52:17.797035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.796921    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:52:17.797606 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.797584    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:52:17.799025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.799001    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"]
Nov 01 20:52:17.869511 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.869476    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj78k\" (UniqueName: \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.869720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.869531    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.869720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.869704    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.970562 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.970519    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.970821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.970579    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj78k\" (UniqueName: \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.970821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.970617    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.971275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.971252    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.973031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.973010    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:17.978229 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:17.978203    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj78k\" (UniqueName: \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\") pod \"maas-api-revoke-sub-e2547622-wl55v\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:18.106447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:18.106359    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:18.379381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:18.379352    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"]
Nov 01 20:52:18.380552 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:52:18.380516    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5971f8_5685_44db_86d0_5f27781ead0d.slice/crio-be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8 WatchSource:0}: Error finding container be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8: Status 404 returned error can't find the container with id be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8
Nov 01 20:52:18.568253 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:18.568222    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v" event={"ID":"8a5971f8-5685-44db-86d0-5f27781ead0d","Type":"ContainerStarted","Data":"be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8"}
Nov 01 20:52:19.575513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:19.575481    5041 generic.go:356] "Generic (PLEG): container finished" podID="8a5971f8-5685-44db-86d0-5f27781ead0d" containerID="aa993b2acb6be0f56372b7feebeecf89b128178f1f33ea46fa938ae038913576" exitCode=0
Nov 01 20:52:19.575953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:19.575569    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v" event={"ID":"8a5971f8-5685-44db-86d0-5f27781ead0d","Type":"ContainerDied","Data":"aa993b2acb6be0f56372b7feebeecf89b128178f1f33ea46fa938ae038913576"}
Nov 01 20:52:20.725861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.725828    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:20.798381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.798346    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\") pod \"8a5971f8-5685-44db-86d0-5f27781ead0d\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") "
Nov 01 20:52:20.798560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.798430    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\" (UniqueName: \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\") pod \"8a5971f8-5685-44db-86d0-5f27781ead0d\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") "
Nov 01 20:52:20.798560 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.798469    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\") pod \"8a5971f8-5685-44db-86d0-5f27781ead0d\" (UID: \"8a5971f8-5685-44db-86d0-5f27781ead0d\") "
Nov 01 20:52:20.798720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.798696    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca" pod "8a5971f8-5685-44db-86d0-5f27781ead0d" (UID: "8a5971f8-5685-44db-86d0-5f27781ead0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:20.798912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.798887    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:20.800547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.800520    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca" pod "8a5971f8-5685-44db-86d0-5f27781ead0d" (UID: "8a5971f8-5685-44db-86d0-5f27781ead0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:52:20.800650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.800626    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k" pod "8a5971f8-5685-44db-86d0-5f27781ead0d" (UID: "8a5971f8-5685-44db-86d0-5f27781ead0d"). InnerVolumeSpecName "kube-api-access-qj78k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:20.899886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.899804    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj78k\" (UniqueName: \"kubernetes.io/projected/8a5971f8-5685-44db-86d0-5f27781ead0d-kube-api-access-qj78k\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:20.899886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:20.899835    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5971f8-5685-44db-86d0-5f27781ead0d-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:21.587570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:21.587535    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"
Nov 01 20:52:21.587771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:21.587567    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v" event={"ID":"8a5971f8-5685-44db-86d0-5f27781ead0d","Type":"ContainerDied","Data":"be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8"}
Nov 01 20:52:21.587771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:21.587622    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be34b60850387961a5adb44c01da8f869fa09024b217475f84a8ead23fcc92a8"
Nov 01 20:52:28.543859 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:52:28.543819    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:52:36.954663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.954623    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"]
Nov 01 20:52:36.955408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.955384    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8a5971f8-5685-44db-86d0-5f27781ead0d" containerName="revoke-keys"
Nov 01 20:52:36.955464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.955414    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5971f8-5685-44db-86d0-5f27781ead0d" containerName="revoke-keys"
Nov 01 20:52:36.955579 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.955565    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5971f8-5685-44db-86d0-5f27781ead0d" containerName="revoke-keys"
Nov 01 20:52:36.959241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.959225    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:36.963619 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.963590    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"]
Nov 01 20:52:36.966474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.965083    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:52:36.966474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.965122    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:52:36.966474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.965142    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:52:36.966474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:36.965184    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:52:37.002231 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.002202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.002398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.002256    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.002442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.002391    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmgw\" (UniqueName: \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.102875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.102839    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.103048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.102956    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmgw\" (UniqueName: \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.103048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.103002    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.107831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.103976    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.107831 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.106569    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.112337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.111530    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmgw\" (UniqueName: \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\") pod \"maas-api-revoke-sub-c1e271a9-wgfsf\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.277158 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.277070    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:37.549506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.549475    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"]
Nov 01 20:52:37.551655 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:52:37.551625    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60888dd8_e975_4d72_bddd_f5b283612a81.slice/crio-88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0 WatchSource:0}: Error finding container 88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0: Status 404 returned error can't find the container with id 88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0
Nov 01 20:52:37.728864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.728824    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf" event={"ID":"60888dd8-e975-4d72-bddd-f5b283612a81","Type":"ContainerStarted","Data":"88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0"}
Nov 01 20:52:37.859773 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.859750    5041 scope.go:122] "RemoveContainer" containerID="443614aaaa66a0a11b7da444ebea37d6d7ee75a34bcd5ce74ffbd5b13d3d92f6"
Nov 01 20:52:37.870449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:37.870423    5041 scope.go:122] "RemoveContainer" containerID="4638da3a259d0f8f0eb9a1d7819894a87b0e4d2c3daa78de065479a44d822f49"
Nov 01 20:52:38.735326 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:38.735290    5041 generic.go:356] "Generic (PLEG): container finished" podID="60888dd8-e975-4d72-bddd-f5b283612a81" containerID="5464ba4976fea485ef0fbb6f14841d33d6862b93e861ebc0668fabc710027322" exitCode=0
Nov 01 20:52:38.735766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:38.735374    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf" event={"ID":"60888dd8-e975-4d72-bddd-f5b283612a81","Type":"ContainerDied","Data":"5464ba4976fea485ef0fbb6f14841d33d6862b93e861ebc0668fabc710027322"}
Nov 01 20:52:39.886435 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.886407    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:39.926583 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.926548    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\") pod \"60888dd8-e975-4d72-bddd-f5b283612a81\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") "
Nov 01 20:52:39.926755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.926593    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\") pod \"60888dd8-e975-4d72-bddd-f5b283612a81\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") "
Nov 01 20:52:39.926755 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.926632    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\" (UniqueName: \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\") pod \"60888dd8-e975-4d72-bddd-f5b283612a81\" (UID: \"60888dd8-e975-4d72-bddd-f5b283612a81\") "
Nov 01 20:52:39.927101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.927070    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca" pod "60888dd8-e975-4d72-bddd-f5b283612a81" (UID: "60888dd8-e975-4d72-bddd-f5b283612a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:39.928861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.928830    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca" pod "60888dd8-e975-4d72-bddd-f5b283612a81" (UID: "60888dd8-e975-4d72-bddd-f5b283612a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:52:39.928994 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:39.928932    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw" pod "60888dd8-e975-4d72-bddd-f5b283612a81" (UID: "60888dd8-e975-4d72-bddd-f5b283612a81"). InnerVolumeSpecName "kube-api-access-jcmgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:40.027692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.027604    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:40.027692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.027634    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60888dd8-e975-4d72-bddd-f5b283612a81-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:40.027692 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.027646    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmgw\" (UniqueName: \"kubernetes.io/projected/60888dd8-e975-4d72-bddd-f5b283612a81-kube-api-access-jcmgw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:40.747087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.746998    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"
Nov 01 20:52:40.747087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.747033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf" event={"ID":"60888dd8-e975-4d72-bddd-f5b283612a81","Type":"ContainerDied","Data":"88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0"}
Nov 01 20:52:40.747087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:40.747085    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88523dd8335686c1825fe29917f9ccce015c794751d1a25e5154853bb81850c0"
Nov 01 20:52:47.765393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.765355    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"]
Nov 01 20:52:47.765875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.765858    5041 cpu_manager.go:414] "containerMap: removing container" podUID="60888dd8-e975-4d72-bddd-f5b283612a81" containerName="revoke-keys"
Nov 01 20:52:47.765875 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.765875    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60888dd8-e975-4d72-bddd-f5b283612a81" containerName="revoke-keys"
Nov 01 20:52:47.766035 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.765965    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="60888dd8-e975-4d72-bddd-f5b283612a81" containerName="revoke-keys"
Nov 01 20:52:47.769349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.769323    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.774855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.774825    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:52:47.775038 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.774884    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:52:47.775212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.775024    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:52:47.778536 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.778506    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"]
Nov 01 20:52:47.779227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.778883    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:52:47.798315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.798284    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8gh\" (UniqueName: \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.798506 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.798353    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.798571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.798543    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.900050 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.900021    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.900219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.900100    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.900219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.900188    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8gh\" (UniqueName: \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.900747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.900720    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.902481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.902460    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:47.908059 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:47.908029    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8gh\" (UniqueName: \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\") pod \"maas-api-revoke-sub-ec843018-xmtxd\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:48.092826 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:48.092772    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:48.369357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:48.369335    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"]
Nov 01 20:52:48.371640 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:52:48.371606    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d19fcb_d894_461c_851d_27838ecfbb4e.slice/crio-99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895 WatchSource:0}: Error finding container 99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895: Status 404 returned error can't find the container with id 99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895
Nov 01 20:52:48.797393 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:48.797354    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd" event={"ID":"97d19fcb-d894-461c-851d-27838ecfbb4e","Type":"ContainerStarted","Data":"1062ab28a68af30ae438b84d08e61a4cdd6a152c7f23498332a60bcdc4f5304b"}
Nov 01 20:52:48.798020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:48.797993    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd" event={"ID":"97d19fcb-d894-461c-851d-27838ecfbb4e","Type":"ContainerStarted","Data":"99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895"}
Nov 01 20:52:48.814420 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:48.814357    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd" podStartSLOduration=1.8143399850000002 podStartE2EDuration="1.814339985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:52:48.812671339 +0000 UTC m=+3256.533750026" watchObservedRunningTime="2026-11-01 20:52:48.814339985 +0000 UTC m=+3256.535418664"
Nov 01 20:52:49.818891 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:49.818850    5041 generic.go:356] "Generic (PLEG): container finished" podID="97d19fcb-d894-461c-851d-27838ecfbb4e" containerID="1062ab28a68af30ae438b84d08e61a4cdd6a152c7f23498332a60bcdc4f5304b" exitCode=0
Nov 01 20:52:49.819365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:49.818930    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd" event={"ID":"97d19fcb-d894-461c-851d-27838ecfbb4e","Type":"ContainerDied","Data":"1062ab28a68af30ae438b84d08e61a4cdd6a152c7f23498332a60bcdc4f5304b"}
Nov 01 20:52:51.290256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.289926    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:51.336753 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.336722    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\" (UniqueName: \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\") pod \"97d19fcb-d894-461c-851d-27838ecfbb4e\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") "
Nov 01 20:52:51.337026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.336802    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\") pod \"97d19fcb-d894-461c-851d-27838ecfbb4e\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") "
Nov 01 20:52:51.337026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.336849    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\") pod \"97d19fcb-d894-461c-851d-27838ecfbb4e\" (UID: \"97d19fcb-d894-461c-851d-27838ecfbb4e\") "
Nov 01 20:52:51.337274 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.337239    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca" pod "97d19fcb-d894-461c-851d-27838ecfbb4e" (UID: "97d19fcb-d894-461c-851d-27838ecfbb4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:52:51.339013 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.338987    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh" pod "97d19fcb-d894-461c-851d-27838ecfbb4e" (UID: "97d19fcb-d894-461c-851d-27838ecfbb4e"). InnerVolumeSpecName "kube-api-access-qs8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:52:51.339115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.339037    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca" pod "97d19fcb-d894-461c-851d-27838ecfbb4e" (UID: "97d19fcb-d894-461c-851d-27838ecfbb4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:52:51.438028 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.437995    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs8gh\" (UniqueName: \"kubernetes.io/projected/97d19fcb-d894-461c-851d-27838ecfbb4e-kube-api-access-qs8gh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:51.438282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.438256    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:51.438282 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.438280    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d19fcb-d894-461c-851d-27838ecfbb4e-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:52:51.833083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.833052    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"
Nov 01 20:52:51.833264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.833052    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd" event={"ID":"97d19fcb-d894-461c-851d-27838ecfbb4e","Type":"ContainerDied","Data":"99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895"}
Nov 01 20:52:51.833264 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:52:51.833171    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99489af59e7150e25999c1df0698b6a89a6d7d2e44c69f6509c17f3e622b6895"
Nov 01 20:52:58.541641 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:52:58.541605    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:53:10.661313 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.661276    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"]
Nov 01 20:53:10.662016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.661994    5041 cpu_manager.go:414] "containerMap: removing container" podUID="97d19fcb-d894-461c-851d-27838ecfbb4e" containerName="revoke-keys"
Nov 01 20:53:10.662118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.662018    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d19fcb-d894-461c-851d-27838ecfbb4e" containerName="revoke-keys"
Nov 01 20:53:10.662177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.662130    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d19fcb-d894-461c-851d-27838ecfbb4e" containerName="revoke-keys"
Nov 01 20:53:10.665411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.665389    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.669768 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.669744    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:53:10.670025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.670008    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:53:10.670200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.670177    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:53:10.670315 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.670261    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:53:10.671553 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.671533    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"]
Nov 01 20:53:10.770240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.770209    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.770430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.770246    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx62\" (UniqueName: \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.770430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.770278    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.871027 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.870993    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.871200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.871033    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nx62\" (UniqueName: \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.871200 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.871076    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.871695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.871667    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.873496 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.873474    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.879829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.879802    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nx62\" (UniqueName: \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\") pod \"maas-api-revoke-sub-046e1343-2khwh\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:10.983760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:10.983678    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:11.258548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:11.258523    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"]
Nov 01 20:53:11.988516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:11.988420    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh" event={"ID":"c886f375-21ec-4057-9eb2-32e3fa43d3d9","Type":"ContainerStarted","Data":"0052924790e5d2b7091663bd3de1eb8a92d61c2b927bede08dc0b065a5825d2e"}
Nov 01 20:53:11.988516 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:11.988483    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh" event={"ID":"c886f375-21ec-4057-9eb2-32e3fa43d3d9","Type":"ContainerStarted","Data":"d6b5546aa1d9d367dc054f2920f58f69dad73d5f2cadd72a0443b26ee70d8003"}
Nov 01 20:53:12.004007 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:12.003955    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh" podStartSLOduration=2.003941549 podStartE2EDuration="2.003941549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:12.00285532 +0000 UTC m=+3279.723933998" watchObservedRunningTime="2026-11-01 20:53:12.003941549 +0000 UTC m=+3279.725020226"
Nov 01 20:53:12.995115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:12.995034    5041 generic.go:356] "Generic (PLEG): container finished" podID="c886f375-21ec-4057-9eb2-32e3fa43d3d9" containerID="0052924790e5d2b7091663bd3de1eb8a92d61c2b927bede08dc0b065a5825d2e" exitCode=0
Nov 01 20:53:12.995534 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:12.995118    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh" event={"ID":"c886f375-21ec-4057-9eb2-32e3fa43d3d9","Type":"ContainerDied","Data":"0052924790e5d2b7091663bd3de1eb8a92d61c2b927bede08dc0b065a5825d2e"}
Nov 01 20:53:14.154715 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.154685    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:14.207309 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.207279    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\") pod \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") "
Nov 01 20:53:14.207472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.207384    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\") pod \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") "
Nov 01 20:53:14.207472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.207408    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\" (UniqueName: \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\") pod \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\" (UID: \"c886f375-21ec-4057-9eb2-32e3fa43d3d9\") "
Nov 01 20:53:14.207805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.207756    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca" pod "c886f375-21ec-4057-9eb2-32e3fa43d3d9" (UID: "c886f375-21ec-4057-9eb2-32e3fa43d3d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:53:14.209385 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.209353    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca" pod "c886f375-21ec-4057-9eb2-32e3fa43d3d9" (UID: "c886f375-21ec-4057-9eb2-32e3fa43d3d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:53:14.209546 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.209519    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62" pod "c886f375-21ec-4057-9eb2-32e3fa43d3d9" (UID: "c886f375-21ec-4057-9eb2-32e3fa43d3d9"). InnerVolumeSpecName "kube-api-access-7nx62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:53:14.308298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.308266    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:14.308298 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.308297    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886f375-21ec-4057-9eb2-32e3fa43d3d9-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:14.308523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:14.308308    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nx62\" (UniqueName: \"kubernetes.io/projected/c886f375-21ec-4057-9eb2-32e3fa43d3d9-kube-api-access-7nx62\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:15.007224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:15.007191    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"
Nov 01 20:53:15.007414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:15.007186    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh" event={"ID":"c886f375-21ec-4057-9eb2-32e3fa43d3d9","Type":"ContainerDied","Data":"d6b5546aa1d9d367dc054f2920f58f69dad73d5f2cadd72a0443b26ee70d8003"}
Nov 01 20:53:15.007414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:15.007322    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b5546aa1d9d367dc054f2920f58f69dad73d5f2cadd72a0443b26ee70d8003"
Nov 01 20:53:20.853523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.853486    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"]
Nov 01 20:53:20.854185 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.854164    5041 cpu_manager.go:414] "containerMap: removing container" podUID="c886f375-21ec-4057-9eb2-32e3fa43d3d9" containerName="revoke-keys"
Nov 01 20:53:20.854285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.854186    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c886f375-21ec-4057-9eb2-32e3fa43d3d9" containerName="revoke-keys"
Nov 01 20:53:20.854368 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.854354    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="c886f375-21ec-4057-9eb2-32e3fa43d3d9" containerName="revoke-keys"
Nov 01 20:53:20.857529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.857505    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:20.862061 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.862021    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:53:20.862177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.862146    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:53:20.862328 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.862309    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:53:20.862428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.862391    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:53:20.864605 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.864577    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"]
Nov 01 20:53:20.969276 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.969230    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kbs\" (UniqueName: \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:20.969450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.969291    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:20.969450 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:20.969382    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.070369 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.070333    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kbs\" (UniqueName: \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.070547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.070377    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.070547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.070397    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.071045 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.071018    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.072795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.072761    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.077695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.077674    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kbs\" (UniqueName: \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\") pod \"maas-api-revoke-sub-d3fba056-b99cn\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.177772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.177694    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:21.450921 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:21.450891    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"]
Nov 01 20:53:21.452983 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:53:21.452957    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d6dbcc_ab32_40d2_a9a4_198113213d30.slice/crio-7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f WatchSource:0}: Error finding container 7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f: Status 404 returned error can't find the container with id 7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f
Nov 01 20:53:22.047947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:22.047908    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn" event={"ID":"f1d6dbcc-ab32-40d2-a9a4-198113213d30","Type":"ContainerStarted","Data":"d11004452919eabb204586ddf3c56aafff72c4724738ea078b0f869146a694f5"}
Nov 01 20:53:22.048487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:22.047971    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn" event={"ID":"f1d6dbcc-ab32-40d2-a9a4-198113213d30","Type":"ContainerStarted","Data":"7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f"}
Nov 01 20:53:22.065321 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:22.065257    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn" podStartSLOduration=2.065239274 podStartE2EDuration="2.065239274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:22.062294542 +0000 UTC m=+3289.783373218" watchObservedRunningTime="2026-11-01 20:53:22.065239274 +0000 UTC m=+3289.786317951"
Nov 01 20:53:23.088110 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:23.088057    5041 generic.go:356] "Generic (PLEG): container finished" podID="f1d6dbcc-ab32-40d2-a9a4-198113213d30" containerID="d11004452919eabb204586ddf3c56aafff72c4724738ea078b0f869146a694f5" exitCode=0
Nov 01 20:53:23.088829 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:23.088799    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn" event={"ID":"f1d6dbcc-ab32-40d2-a9a4-198113213d30","Type":"ContainerDied","Data":"d11004452919eabb204586ddf3c56aafff72c4724738ea078b0f869146a694f5"}
Nov 01 20:53:24.400381 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.400351    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:24.430658 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.430628    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\") pod \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") "
Nov 01 20:53:24.431056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.431032    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\" (UniqueName: \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\") pod \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") "
Nov 01 20:53:24.431249 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.431232    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\") pod \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\" (UID: \"f1d6dbcc-ab32-40d2-a9a4-198113213d30\") "
Nov 01 20:53:24.431624 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.431583    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca" pod "f1d6dbcc-ab32-40d2-a9a4-198113213d30" (UID: "f1d6dbcc-ab32-40d2-a9a4-198113213d30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:53:24.434814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.433761    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca" pod "f1d6dbcc-ab32-40d2-a9a4-198113213d30" (UID: "f1d6dbcc-ab32-40d2-a9a4-198113213d30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:53:24.434814 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.434058    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs" pod "f1d6dbcc-ab32-40d2-a9a4-198113213d30" (UID: "f1d6dbcc-ab32-40d2-a9a4-198113213d30"). InnerVolumeSpecName "kube-api-access-c2kbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:53:24.532469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.532419    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:24.532469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.532456    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2kbs\" (UniqueName: \"kubernetes.io/projected/f1d6dbcc-ab32-40d2-a9a4-198113213d30-kube-api-access-c2kbs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:24.532469 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:24.532465    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d6dbcc-ab32-40d2-a9a4-198113213d30-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:25.114568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:25.114537    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"
Nov 01 20:53:25.114568 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:25.114554    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn" event={"ID":"f1d6dbcc-ab32-40d2-a9a4-198113213d30","Type":"ContainerDied","Data":"7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f"}
Nov 01 20:53:25.114816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:25.114597    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f8a8591009443c116fc035bc2986737c2ccf0ecf5276f9437e0572a2ac9e06f"
Nov 01 20:53:28.540487 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:53:28.540451    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:53:33.122241 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:33.122208    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:53:33.137095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:33.137072    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:53:37.896062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:37.896023    5041 scope.go:122] "RemoveContainer" containerID="884520a8ccbdb32c0af9642059266f9b25047f0e7d63003aca9690abc192a015"
Nov 01 20:53:41.275582 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.275533    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"]
Nov 01 20:53:41.276068 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.276052    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f1d6dbcc-ab32-40d2-a9a4-198113213d30" containerName="revoke-keys"
Nov 01 20:53:41.276133 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.276069    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d6dbcc-ab32-40d2-a9a4-198113213d30" containerName="revoke-keys"
Nov 01 20:53:41.276173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.276146    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d6dbcc-ab32-40d2-a9a4-198113213d30" containerName="revoke-keys"
Nov 01 20:53:41.280754 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.280736    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.285533 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.285502    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:53:41.285680 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.285605    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:53:41.285745 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.285695    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:53:41.285859 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.285761    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:53:41.286414 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.286393    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"]
Nov 01 20:53:41.427152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.427110    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.427352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.427233    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.427352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.427327    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.528716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.528622    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.529043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.528749    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.529043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.528826    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.529398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.529379    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.531189 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.531167    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.537760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.537735    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\") pod \"maas-api-revoke-sub-ae594979-j74rj\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.598744 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.598702    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:41.873610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:41.873582    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"]
Nov 01 20:53:41.875614 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:53:41.875587    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1883233_5c9d_40ca_8b47_8b64efe7ccfd.slice/crio-0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc WatchSource:0}: Error finding container 0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc: Status 404 returned error can't find the container with id 0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc
Nov 01 20:53:42.229173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:42.229133    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj" event={"ID":"d1883233-5c9d-40ca-8b47-8b64efe7ccfd","Type":"ContainerStarted","Data":"3ee0aea7bcdc20aec2c495995a88165140a61b77c4fffc821ffcc101ff79045c"}
Nov 01 20:53:42.229360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:42.229200    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj" event={"ID":"d1883233-5c9d-40ca-8b47-8b64efe7ccfd","Type":"ContainerStarted","Data":"0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc"}
Nov 01 20:53:42.246991 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:42.246939    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj" podStartSLOduration=1.246925396 podStartE2EDuration="1.246925396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:53:42.24423918 +0000 UTC m=+3309.965317860" watchObservedRunningTime="2026-11-01 20:53:42.246925396 +0000 UTC m=+3309.968004072"
Nov 01 20:53:43.237201 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:43.237163    5041 generic.go:356] "Generic (PLEG): container finished" podID="d1883233-5c9d-40ca-8b47-8b64efe7ccfd" containerID="3ee0aea7bcdc20aec2c495995a88165140a61b77c4fffc821ffcc101ff79045c" exitCode=0
Nov 01 20:53:43.237596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:43.237248    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj" event={"ID":"d1883233-5c9d-40ca-8b47-8b64efe7ccfd","Type":"ContainerDied","Data":"3ee0aea7bcdc20aec2c495995a88165140a61b77c4fffc821ffcc101ff79045c"}
Nov 01 20:53:44.382063 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.382038    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:44.458488 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.458456    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\") pod \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") "
Nov 01 20:53:44.458666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.458546    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\") pod \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") "
Nov 01 20:53:44.458666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.458617    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\") pod \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\" (UID: \"d1883233-5c9d-40ca-8b47-8b64efe7ccfd\") "
Nov 01 20:53:44.458993 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.458970    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca" pod "d1883233-5c9d-40ca-8b47-8b64efe7ccfd" (UID: "d1883233-5c9d-40ca-8b47-8b64efe7ccfd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:53:44.460593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.460568    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca" pod "d1883233-5c9d-40ca-8b47-8b64efe7ccfd" (UID: "d1883233-5c9d-40ca-8b47-8b64efe7ccfd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:53:44.460593 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.460579    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps" pod "d1883233-5c9d-40ca-8b47-8b64efe7ccfd" (UID: "d1883233-5c9d-40ca-8b47-8b64efe7ccfd"). InnerVolumeSpecName "kube-api-access-4fgps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:53:44.559915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.559885    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:44.559915 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.559916    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgps\" (UniqueName: \"kubernetes.io/projected/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-kube-api-access-4fgps\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:44.560125 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:44.559930    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1883233-5c9d-40ca-8b47-8b64efe7ccfd-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:53:45.250173 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:45.250139    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"
Nov 01 20:53:45.250357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:45.250171    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj" event={"ID":"d1883233-5c9d-40ca-8b47-8b64efe7ccfd","Type":"ContainerDied","Data":"0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc"}
Nov 01 20:53:45.250357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:45.250224    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5872809753f45b874e1749cc63e68e316b24de115454925156dd30394c68fc"
Nov 01 20:53:57.438731 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.438690    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"]
Nov 01 20:53:57.439273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.439249    5041 cpu_manager.go:414] "containerMap: removing container" podUID="d1883233-5c9d-40ca-8b47-8b64efe7ccfd" containerName="revoke-keys"
Nov 01 20:53:57.439458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.439440    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1883233-5c9d-40ca-8b47-8b64efe7ccfd" containerName="revoke-keys"
Nov 01 20:53:57.439759 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.439739    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1883233-5c9d-40ca-8b47-8b64efe7ccfd" containerName="revoke-keys"
Nov 01 20:53:57.443389 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.443373    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.447212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.446886    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:53:57.447212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.446895    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:53:57.447212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.446964    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:53:57.447769 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.447750    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:53:57.449671 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.449642    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"]
Nov 01 20:53:57.527653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.527619    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sgh\" (UniqueName: \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.527842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.527740    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.527909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.527891    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.628403 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.628367    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sgh\" (UniqueName: \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.628571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.628454    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.628571 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.628486    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.629150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.629117    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.630861 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.630837    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.640085 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.640063    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sgh\" (UniqueName: \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\") pod \"maas-api-revoke-sub-a4aef7d7-gfqwf\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:57.760695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:57.760620    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:53:58.033840 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:58.033811    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"]
Nov 01 20:53:58.035351 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:53:58.035322    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db9ad78_79f5_4b21_a383_ada349450d85.slice/crio-606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986 WatchSource:0}: Error finding container 606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986: Status 404 returned error can't find the container with id 606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986
Nov 01 20:53:58.340743 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:58.340707    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf" event={"ID":"3db9ad78-79f5-4b21-a383-ada349450d85","Type":"ContainerStarted","Data":"606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986"}
Nov 01 20:53:58.539193 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:53:58.539161    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:53:59.350217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:59.350173    5041 generic.go:356] "Generic (PLEG): container finished" podID="3db9ad78-79f5-4b21-a383-ada349450d85" containerID="878b174eb2636b038829f2052c2f017c644fabe5915019b9d3212e710916b75c" exitCode=0
Nov 01 20:53:59.350413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:53:59.350311    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf" event={"ID":"3db9ad78-79f5-4b21-a383-ada349450d85","Type":"ContainerDied","Data":"878b174eb2636b038829f2052c2f017c644fabe5915019b9d3212e710916b75c"}
Nov 01 20:54:00.510889 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.510863    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:54:00.557938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.557901    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\" (UniqueName: \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\") pod \"3db9ad78-79f5-4b21-a383-ada349450d85\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") "
Nov 01 20:54:00.558104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.557967    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\") pod \"3db9ad78-79f5-4b21-a383-ada349450d85\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") "
Nov 01 20:54:00.558176 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.558159    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\") pod \"3db9ad78-79f5-4b21-a383-ada349450d85\" (UID: \"3db9ad78-79f5-4b21-a383-ada349450d85\") "
Nov 01 20:54:00.558322 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.558298    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca" pod "3db9ad78-79f5-4b21-a383-ada349450d85" (UID: "3db9ad78-79f5-4b21-a383-ada349450d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:00.558525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.558505    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:00.560123 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.560093    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh" pod "3db9ad78-79f5-4b21-a383-ada349450d85" (UID: "3db9ad78-79f5-4b21-a383-ada349450d85"). InnerVolumeSpecName "kube-api-access-k9sgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:00.560258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.560178    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca" pod "3db9ad78-79f5-4b21-a383-ada349450d85" (UID: "3db9ad78-79f5-4b21-a383-ada349450d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:54:00.659430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.659349    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db9ad78-79f5-4b21-a383-ada349450d85-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:00.659430 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:00.659379    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9sgh\" (UniqueName: \"kubernetes.io/projected/3db9ad78-79f5-4b21-a383-ada349450d85-kube-api-access-k9sgh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:01.364633 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:01.364603    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"
Nov 01 20:54:01.364828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:01.364603    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf" event={"ID":"3db9ad78-79f5-4b21-a383-ada349450d85","Type":"ContainerDied","Data":"606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986"}
Nov 01 20:54:01.364828 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:01.364729    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606bc9008d257ea534b68d6d20f27e061b10aae63d7f860e0764ad61887cf986"
Nov 01 20:54:12.728005 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.727966    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"]
Nov 01 20:54:12.728719 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.728694    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3db9ad78-79f5-4b21-a383-ada349450d85" containerName="revoke-keys"
Nov 01 20:54:12.728842 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.728723    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db9ad78-79f5-4b21-a383-ada349450d85" containerName="revoke-keys"
Nov 01 20:54:12.728912 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.728891    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db9ad78-79f5-4b21-a383-ada349450d85" containerName="revoke-keys"
Nov 01 20:54:12.732612 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.732589    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"]
Nov 01 20:54:12.732762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.732707    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.737489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.737458    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:54:12.737638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.737569    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:54:12.737937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.737740    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:54:12.737937 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.737929    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:54:12.828695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.828650    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.828978 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.828951    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwqn\" (UniqueName: \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.829112 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.829055    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.930462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.930422    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.930666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.930473    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwqn\" (UniqueName: \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.930666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.930557    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.931219 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.931195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.933208 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.933183    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:12.938314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:12.938286    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwqn\" (UniqueName: \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\") pod \"maas-api-revoke-sub-79805ca9-7shzh\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:13.109837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:13.109803    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:13.383365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:13.383339    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"]
Nov 01 20:54:13.384524 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:54:13.384498    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf076a401_64f0_4832_aa5f_bb3174eb1807.slice/crio-fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d WatchSource:0}: Error finding container fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d: Status 404 returned error can't find the container with id fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d
Nov 01 20:54:13.438521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:13.438492    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh" event={"ID":"f076a401-64f0-4832-aa5f-bb3174eb1807","Type":"ContainerStarted","Data":"fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d"}
Nov 01 20:54:14.445037 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:14.444998    5041 generic.go:356] "Generic (PLEG): container finished" podID="f076a401-64f0-4832-aa5f-bb3174eb1807" containerID="df0bdec866edac8729833f03c9c5c0fd7a5c4f4528b373546b896487947b2388" exitCode=0
Nov 01 20:54:14.445447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:14.445033    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh" event={"ID":"f076a401-64f0-4832-aa5f-bb3174eb1807","Type":"ContainerDied","Data":"df0bdec866edac8729833f03c9c5c0fd7a5c4f4528b373546b896487947b2388"}
Nov 01 20:54:15.600799 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.600755    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:15.759227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.759148    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\" (UniqueName: \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\") pod \"f076a401-64f0-4832-aa5f-bb3174eb1807\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") "
Nov 01 20:54:15.759227 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.759204    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\") pod \"f076a401-64f0-4832-aa5f-bb3174eb1807\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") "
Nov 01 20:54:15.759480 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.759336    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\") pod \"f076a401-64f0-4832-aa5f-bb3174eb1807\" (UID: \"f076a401-64f0-4832-aa5f-bb3174eb1807\") "
Nov 01 20:54:15.759632 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.759608    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca" pod "f076a401-64f0-4832-aa5f-bb3174eb1807" (UID: "f076a401-64f0-4832-aa5f-bb3174eb1807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:15.761337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.761312    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca" pod "f076a401-64f0-4832-aa5f-bb3174eb1807" (UID: "f076a401-64f0-4832-aa5f-bb3174eb1807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:54:15.761544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.761514    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn" pod "f076a401-64f0-4832-aa5f-bb3174eb1807" (UID: "f076a401-64f0-4832-aa5f-bb3174eb1807"). InnerVolumeSpecName "kube-api-access-2lwqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:15.860233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.860200    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:15.860233 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.860231    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwqn\" (UniqueName: \"kubernetes.io/projected/f076a401-64f0-4832-aa5f-bb3174eb1807-kube-api-access-2lwqn\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:15.860445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:15.860245    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f076a401-64f0-4832-aa5f-bb3174eb1807-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:16.456916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:16.456891    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"
Nov 01 20:54:16.456916 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:16.456900    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh" event={"ID":"f076a401-64f0-4832-aa5f-bb3174eb1807","Type":"ContainerDied","Data":"fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d"}
Nov 01 20:54:16.457118 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:16.456944    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa7e60298b029395a65c85ffd6da317f444b1ef885e586d7b790046d50096c8d"
Nov 01 20:54:27.220797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.215833    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"]
Nov 01 20:54:27.220797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.216594    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f076a401-64f0-4832-aa5f-bb3174eb1807" containerName="revoke-keys"
Nov 01 20:54:27.220797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.216616    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f076a401-64f0-4832-aa5f-bb3174eb1807" containerName="revoke-keys"
Nov 01 20:54:27.220797 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.216795    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f076a401-64f0-4832-aa5f-bb3174eb1807" containerName="revoke-keys"
Nov 01 20:54:27.233160 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.233128    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"]
Nov 01 20:54:27.233356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.233242    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.238521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.238463    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:54:27.238521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.238482    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:54:27.238521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.238479    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:54:27.238825 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.238542    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:54:27.316638 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.316588    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cx49\" (UniqueName: \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.317016 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.316987    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.317188 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.317161    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.419464 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.419409    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cx49\" (UniqueName: \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.419645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.419551    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.419645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.419587    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.422525 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.422501    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.425457 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.425408    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.429182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.429156    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cx49\" (UniqueName: \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\") pod \"maas-api-revoke-sub-0b7e8edb-94xfn\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.616055 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.616003    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:27.984019 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:27.978074    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"]
Nov 01 20:54:28.534862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:28.534012    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn" event={"ID":"196857ee-6fc6-4864-afa2-a18fcaa6c257","Type":"ContainerStarted","Data":"860d4cfeea49da215b522496d4b666f9f7bc8e537635c32870055b71f2ce3de5"}
Nov 01 20:54:28.534862 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:28.534076    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn" event={"ID":"196857ee-6fc6-4864-afa2-a18fcaa6c257","Type":"ContainerStarted","Data":"71703612e16248099906a063d30ca21f1a46b3be6737dae73037f802368473cf"}
Nov 01 20:54:28.542448 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:54:28.542387    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:54:28.552380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:28.552335    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn" podStartSLOduration=1.5523212389999999 podStartE2EDuration="1.552321239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:54:28.548837474 +0000 UTC m=+3356.269916151" watchObservedRunningTime="2026-11-01 20:54:28.552321239 +0000 UTC m=+3356.273399915"
Nov 01 20:54:29.540819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:29.540757    5041 generic.go:356] "Generic (PLEG): container finished" podID="196857ee-6fc6-4864-afa2-a18fcaa6c257" containerID="860d4cfeea49da215b522496d4b666f9f7bc8e537635c32870055b71f2ce3de5" exitCode=0
Nov 01 20:54:29.541291 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:29.540845    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn" event={"ID":"196857ee-6fc6-4864-afa2-a18fcaa6c257","Type":"ContainerDied","Data":"860d4cfeea49da215b522496d4b666f9f7bc8e537635c32870055b71f2ce3de5"}
Nov 01 20:54:30.693421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.693399    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:30.758250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.758202    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\") pod \"196857ee-6fc6-4864-afa2-a18fcaa6c257\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") "
Nov 01 20:54:30.758466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.758281    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\" (UniqueName: \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\") pod \"196857ee-6fc6-4864-afa2-a18fcaa6c257\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") "
Nov 01 20:54:30.758466 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.758395    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\") pod \"196857ee-6fc6-4864-afa2-a18fcaa6c257\" (UID: \"196857ee-6fc6-4864-afa2-a18fcaa6c257\") "
Nov 01 20:54:30.758677 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.758647    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca" pod "196857ee-6fc6-4864-afa2-a18fcaa6c257" (UID: "196857ee-6fc6-4864-afa2-a18fcaa6c257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:30.758804 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.758773    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:30.761084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.761063    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca" pod "196857ee-6fc6-4864-afa2-a18fcaa6c257" (UID: "196857ee-6fc6-4864-afa2-a18fcaa6c257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:54:30.761084 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.761072    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49" pod "196857ee-6fc6-4864-afa2-a18fcaa6c257" (UID: "196857ee-6fc6-4864-afa2-a18fcaa6c257"). InnerVolumeSpecName "kube-api-access-6cx49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:30.860140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.860054    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cx49\" (UniqueName: \"kubernetes.io/projected/196857ee-6fc6-4864-afa2-a18fcaa6c257-kube-api-access-6cx49\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:30.860140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:30.860090    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196857ee-6fc6-4864-afa2-a18fcaa6c257-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:31.552659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:31.552617    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn" event={"ID":"196857ee-6fc6-4864-afa2-a18fcaa6c257","Type":"ContainerDied","Data":"71703612e16248099906a063d30ca21f1a46b3be6737dae73037f802368473cf"}
Nov 01 20:54:31.552659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:31.552651    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"
Nov 01 20:54:31.552922 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:31.552671    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71703612e16248099906a063d30ca21f1a46b3be6737dae73037f802368473cf"
Nov 01 20:54:37.020944 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.020897    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"]
Nov 01 20:54:37.023566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.023534    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7122df3-mpltg"]
Nov 01 20:54:37.824275 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.824243    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"]
Nov 01 20:54:37.824685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.824673    5041 cpu_manager.go:414] "containerMap: removing container" podUID="196857ee-6fc6-4864-afa2-a18fcaa6c257" containerName="revoke-keys"
Nov 01 20:54:37.824734 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.824686    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196857ee-6fc6-4864-afa2-a18fcaa6c257" containerName="revoke-keys"
Nov 01 20:54:37.824802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.824792    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="196857ee-6fc6-4864-afa2-a18fcaa6c257" containerName="revoke-keys"
Nov 01 20:54:37.828029 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.828013    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:37.832741 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.832720    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:54:37.832979 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.832955    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:54:37.833078 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.833046    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:54:37.833168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.833152    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:54:37.835523 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.835498    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"]
Nov 01 20:54:37.925022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.924986    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:37.925211 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.925192    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfld4\" (UniqueName: \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:37.925297 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:37.925282    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.026244 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.026210    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.026634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.026274    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.026634 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.026377    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfld4\" (UniqueName: \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.026989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.026968    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.028685 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.028665    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.034126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.034101    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfld4\" (UniqueName: \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\") pod \"maas-api-revoke-sub-3d4156f1-qwrwg\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.147090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.146998    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:38.423749 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:54:38.423717    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b81cad3_4c7f_43db_9514_d17374e6641a.slice/crio-ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe WatchSource:0}: Error finding container ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe: Status 404 returned error can't find the container with id ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe
Nov 01 20:54:38.428221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.428139    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42638d25-38c5-463a-90f0-e2ff67bf9886" path="/var/lib/kubelet/pods/42638d25-38c5-463a-90f0-e2ff67bf9886/volumes"
Nov 01 20:54:38.428458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.428443    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"]
Nov 01 20:54:38.594806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:38.594747    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg" event={"ID":"2b81cad3-4c7f-43db-9514-d17374e6641a","Type":"ContainerStarted","Data":"ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe"}
Nov 01 20:54:39.602020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:39.601987    5041 generic.go:356] "Generic (PLEG): container finished" podID="2b81cad3-4c7f-43db-9514-d17374e6641a" containerID="481ef2bec2cc12b22390a9ae722b13c1b4bb186dd59f031cbc4acd0958f405fa" exitCode=0
Nov 01 20:54:39.602416 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:39.602067    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg" event={"ID":"2b81cad3-4c7f-43db-9514-d17374e6641a","Type":"ContainerDied","Data":"481ef2bec2cc12b22390a9ae722b13c1b4bb186dd59f031cbc4acd0958f405fa"}
Nov 01 20:54:40.758217 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.758188    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:40.854134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.854103    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\" (UniqueName: \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\") pod \"2b81cad3-4c7f-43db-9514-d17374e6641a\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") "
Nov 01 20:54:40.854310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.854164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\") pod \"2b81cad3-4c7f-43db-9514-d17374e6641a\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") "
Nov 01 20:54:40.854310 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.854200    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\") pod \"2b81cad3-4c7f-43db-9514-d17374e6641a\" (UID: \"2b81cad3-4c7f-43db-9514-d17374e6641a\") "
Nov 01 20:54:40.854596 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.854566    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca" pod "2b81cad3-4c7f-43db-9514-d17374e6641a" (UID: "2b81cad3-4c7f-43db-9514-d17374e6641a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:40.856360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.856330    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4" pod "2b81cad3-4c7f-43db-9514-d17374e6641a" (UID: "2b81cad3-4c7f-43db-9514-d17374e6641a"). InnerVolumeSpecName "kube-api-access-gfld4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:40.856449 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.856388    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca" pod "2b81cad3-4c7f-43db-9514-d17374e6641a" (UID: "2b81cad3-4c7f-43db-9514-d17374e6641a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:54:40.955140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.955101    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfld4\" (UniqueName: \"kubernetes.io/projected/2b81cad3-4c7f-43db-9514-d17374e6641a-kube-api-access-gfld4\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:40.955140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.955133    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:40.955140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:40.955143    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b81cad3-4c7f-43db-9514-d17374e6641a-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:41.615289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:41.615253    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"
Nov 01 20:54:41.615289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:41.615282    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg" event={"ID":"2b81cad3-4c7f-43db-9514-d17374e6641a","Type":"ContainerDied","Data":"ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe"}
Nov 01 20:54:41.615521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:41.615330    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4fb85550f4a3c410a50e9a39f4e5e36e9f5b1ac973d53abd329dca55725ebe"
Nov 01 20:54:54.260696 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.260610    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"]
Nov 01 20:54:54.264367 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.264329    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qwrwg"]
Nov 01 20:54:54.281349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.281314    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"]
Nov 01 20:54:54.281868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.281843    5041 cpu_manager.go:414] "containerMap: removing container" podUID="2b81cad3-4c7f-43db-9514-d17374e6641a" containerName="revoke-keys"
Nov 01 20:54:54.281868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.281861    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b81cad3-4c7f-43db-9514-d17374e6641a" containerName="revoke-keys"
Nov 01 20:54:54.282043 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.281988    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b81cad3-4c7f-43db-9514-d17374e6641a" containerName="revoke-keys"
Nov 01 20:54:54.285481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.285460    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.289113 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.289088    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:54:54.290637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.289651    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:54:54.290637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.289701    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:54:54.290637 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.289829    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:54:54.294507 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.294243    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"]
Nov 01 20:54:54.349481 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.349441    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9js\" (UniqueName: \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.349678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.349616    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.349747 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.349724    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.428346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.428310    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b81cad3-4c7f-43db-9514-d17374e6641a" path="/var/lib/kubelet/pods/2b81cad3-4c7f-43db-9514-d17374e6641a/volumes"
Nov 01 20:54:54.450924 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.450895    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.451065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.450943    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.451065 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.450977    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9js\" (UniqueName: \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.451653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.451627    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.453250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.453227    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.459011 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.458992    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9js\" (UniqueName: \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\") pod \"maas-api-revoke-sub-3d4156f1-jbtrl\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.602471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.602421    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:54.876256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:54.876228    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"]
Nov 01 20:54:54.877865 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:54:54.877842    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ff1d91_3476_42ce_b2f7_deed05e91133.slice/crio-7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f WatchSource:0}: Error finding container 7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f: Status 404 returned error can't find the container with id 7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f
Nov 01 20:54:55.733392 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:55.733356    5041 generic.go:356] "Generic (PLEG): container finished" podID="f4ff1d91-3476-42ce-b2f7-deed05e91133" containerID="d99818d4e5c934c1052f7529c3cb183c09c6662e6873f71c58ed7033c95fe581" exitCode=0
Nov 01 20:54:55.733821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:55.733440    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl" event={"ID":"f4ff1d91-3476-42ce-b2f7-deed05e91133","Type":"ContainerDied","Data":"d99818d4e5c934c1052f7529c3cb183c09c6662e6873f71c58ed7033c95fe581"}
Nov 01 20:54:55.733821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:55.733492    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl" event={"ID":"f4ff1d91-3476-42ce-b2f7-deed05e91133","Type":"ContainerStarted","Data":"7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f"}
Nov 01 20:54:56.892676 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.892650    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:56.973660 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.973625    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\") pod \"f4ff1d91-3476-42ce-b2f7-deed05e91133\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") "
Nov 01 20:54:56.973881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.973673    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\") pod \"f4ff1d91-3476-42ce-b2f7-deed05e91133\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") "
Nov 01 20:54:56.973881 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.973705    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\" (UniqueName: \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\") pod \"f4ff1d91-3476-42ce-b2f7-deed05e91133\" (UID: \"f4ff1d91-3476-42ce-b2f7-deed05e91133\") "
Nov 01 20:54:56.974096 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.974070    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca" pod "f4ff1d91-3476-42ce-b2f7-deed05e91133" (UID: "f4ff1d91-3476-42ce-b2f7-deed05e91133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:54:56.975867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.975832    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca" pod "f4ff1d91-3476-42ce-b2f7-deed05e91133" (UID: "f4ff1d91-3476-42ce-b2f7-deed05e91133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:54:56.975972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:56.975940    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js" pod "f4ff1d91-3476-42ce-b2f7-deed05e91133" (UID: "f4ff1d91-3476-42ce-b2f7-deed05e91133"). InnerVolumeSpecName "kube-api-access-2f9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:54:57.075349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.075311    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:57.075349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.075342    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4ff1d91-3476-42ce-b2f7-deed05e91133-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:57.075349 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.075353    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9js\" (UniqueName: \"kubernetes.io/projected/f4ff1d91-3476-42ce-b2f7-deed05e91133-kube-api-access-2f9js\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:54:57.746604 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.746572    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"
Nov 01 20:54:57.746855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.746609    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl" event={"ID":"f4ff1d91-3476-42ce-b2f7-deed05e91133","Type":"ContainerDied","Data":"7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f"}
Nov 01 20:54:57.746855 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:54:57.746658    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b27ea1b8bdee9a3ab425365c9635fcbe3a151dcb4d922edb0b323da697bb41f"
Nov 01 20:54:58.540699 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:54:58.540666    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:55:16.163357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.163320    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"]
Nov 01 20:55:16.163837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.163822    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f4ff1d91-3476-42ce-b2f7-deed05e91133" containerName="revoke-keys"
Nov 01 20:55:16.163904 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.163838    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ff1d91-3476-42ce-b2f7-deed05e91133" containerName="revoke-keys"
Nov 01 20:55:16.163940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.163925    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ff1d91-3476-42ce-b2f7-deed05e91133" containerName="revoke-keys"
Nov 01 20:55:16.167140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.167120    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.171818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.171772    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:55:16.172661 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.172640    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:55:16.173122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.172642    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:55:16.173122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.172876    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:55:16.174820 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.174757    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"]
Nov 01 20:55:16.322681 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.322646    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqxs\" (UniqueName: \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.322879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.322759    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.322879 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.322858    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.424134 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.424039    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqxs\" (UniqueName: \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.424296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.424194    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.424296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.424280    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.425086 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.425060    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.426876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.426851    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.432075 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.432053    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqxs\" (UniqueName: \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\") pod \"maas-api-revoke-sub-e2fc9c5b-kjlcb\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.494566 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.494530    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:16.768494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.768468    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"]
Nov 01 20:55:16.770175 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:55:16.770143    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f25675d_8378_4940_8175_89ab15524c9b.slice/crio-b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef WatchSource:0}: Error finding container b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef: Status 404 returned error can't find the container with id b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef
Nov 01 20:55:16.889844 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:16.889806    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb" event={"ID":"3f25675d-8378-4940-8175-89ab15524c9b","Type":"ContainerStarted","Data":"b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef"}
Nov 01 20:55:17.897352 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:17.897273    5041 generic.go:356] "Generic (PLEG): container finished" podID="3f25675d-8378-4940-8175-89ab15524c9b" containerID="d1f72d6c218f1a334fd9ff305ac080f59d867e8c35f33a8d0482e1feb4b55423" exitCode=0
Nov 01 20:55:17.897716 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:17.897363    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb" event={"ID":"3f25675d-8378-4940-8175-89ab15524c9b","Type":"ContainerDied","Data":"d1f72d6c218f1a334fd9ff305ac080f59d867e8c35f33a8d0482e1feb4b55423"}
Nov 01 20:55:19.044149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.044125    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:19.153931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.153839    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\" (UniqueName: \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\") pod \"3f25675d-8378-4940-8175-89ab15524c9b\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") "
Nov 01 20:55:19.153931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.153911    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\") pod \"3f25675d-8378-4940-8175-89ab15524c9b\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") "
Nov 01 20:55:19.154190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.153947    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\") pod \"3f25675d-8378-4940-8175-89ab15524c9b\" (UID: \"3f25675d-8378-4940-8175-89ab15524c9b\") "
Nov 01 20:55:19.154269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.154245    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca" pod "3f25675d-8378-4940-8175-89ab15524c9b" (UID: "3f25675d-8378-4940-8175-89ab15524c9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:55:19.154395 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.154377    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:19.156070 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.156041    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca" pod "3f25675d-8378-4940-8175-89ab15524c9b" (UID: "3f25675d-8378-4940-8175-89ab15524c9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:55:19.156184 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.156073    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs" pod "3f25675d-8378-4940-8175-89ab15524c9b" (UID: "3f25675d-8378-4940-8175-89ab15524c9b"). InnerVolumeSpecName "kube-api-access-rlqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:55:19.255095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.255045    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqxs\" (UniqueName: \"kubernetes.io/projected/3f25675d-8378-4940-8175-89ab15524c9b-kube-api-access-rlqxs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:19.255095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.255090    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f25675d-8378-4940-8175-89ab15524c9b-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:19.911424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.911391    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"
Nov 01 20:55:19.911424 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.911402    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb" event={"ID":"3f25675d-8378-4940-8175-89ab15524c9b","Type":"ContainerDied","Data":"b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef"}
Nov 01 20:55:19.911639 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:19.911454    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b980788fd48b7bf76cb0fb04ee63a12a5b7034f72ee7d4e7540705fdf6d7d7ef"
Nov 01 20:55:26.356948 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.356911    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"]
Nov 01 20:55:26.357437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.357383    5041 cpu_manager.go:414] "containerMap: removing container" podUID="3f25675d-8378-4940-8175-89ab15524c9b" containerName="revoke-keys"
Nov 01 20:55:26.357437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.357395    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f25675d-8378-4940-8175-89ab15524c9b" containerName="revoke-keys"
Nov 01 20:55:26.357521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.357509    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f25675d-8378-4940-8175-89ab15524c9b" containerName="revoke-keys"
Nov 01 20:55:26.361014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.360996    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.365025 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.364980    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:55:26.365182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.365133    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:55:26.365304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.365272    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:55:26.365931 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.365899    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:55:26.367319 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.367300    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"]
Nov 01 20:55:26.421003 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.420968    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw4lb\" (UniqueName: \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.421212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.421111    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.421212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.421173    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.521762 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.521716    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw4lb\" (UniqueName: \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.522000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.521871    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.522000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.521907    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.522548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.522520    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.524396 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.524372    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.529273 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.529247    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw4lb\" (UniqueName: \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\") pod \"maas-api-revoke-sub-5a29efb3-4sh8v\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.679174 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.679089    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:26.956720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:26.956688    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"]
Nov 01 20:55:26.957974 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:55:26.957946    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b4f20d_0020_4379_9935_54deb8995804.slice/crio-ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7 WatchSource:0}: Error finding container ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7: Status 404 returned error can't find the container with id ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7
Nov 01 20:55:27.965056 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:27.965011    5041 generic.go:356] "Generic (PLEG): container finished" podID="f5b4f20d-0020-4379-9935-54deb8995804" containerID="ff1a67b9438bc48c4a116b84a56a43abda8d3a5e395dace15d7ca80a1cb3d89a" exitCode=0
Nov 01 20:55:27.965576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:27.965105    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v" event={"ID":"f5b4f20d-0020-4379-9935-54deb8995804","Type":"ContainerDied","Data":"ff1a67b9438bc48c4a116b84a56a43abda8d3a5e395dace15d7ca80a1cb3d89a"}
Nov 01 20:55:27.965576 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:27.965212    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v" event={"ID":"f5b4f20d-0020-4379-9935-54deb8995804","Type":"ContainerStarted","Data":"ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7"}
Nov 01 20:55:28.559601 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:55:28.559443    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:55:29.357802 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.350968    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:29.514805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.512164    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\") pod \"f5b4f20d-0020-4379-9935-54deb8995804\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") "
Nov 01 20:55:29.514805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.512272    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\") pod \"f5b4f20d-0020-4379-9935-54deb8995804\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") "
Nov 01 20:55:29.514805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.512389    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\" (UniqueName: \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\") pod \"f5b4f20d-0020-4379-9935-54deb8995804\" (UID: \"f5b4f20d-0020-4379-9935-54deb8995804\") "
Nov 01 20:55:29.514805 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.514577    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca" pod "f5b4f20d-0020-4379-9935-54deb8995804" (UID: "f5b4f20d-0020-4379-9935-54deb8995804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:55:29.523152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.523123    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca" pod "f5b4f20d-0020-4379-9935-54deb8995804" (UID: "f5b4f20d-0020-4379-9935-54deb8995804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:55:29.523307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.523185    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb" pod "f5b4f20d-0020-4379-9935-54deb8995804" (UID: "f5b4f20d-0020-4379-9935-54deb8995804"). InnerVolumeSpecName "kube-api-access-sw4lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:55:29.620673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.620505    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:29.620673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.620561    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b4f20d-0020-4379-9935-54deb8995804-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:29.620673 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:29.620580    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw4lb\" (UniqueName: \"kubernetes.io/projected/f5b4f20d-0020-4379-9935-54deb8995804-kube-api-access-sw4lb\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:30.016314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:30.016230    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"
Nov 01 20:55:30.016314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:30.016262    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v" event={"ID":"f5b4f20d-0020-4379-9935-54deb8995804","Type":"ContainerDied","Data":"ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7"}
Nov 01 20:55:30.016314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:30.016309    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea71987bf5f93ec96605a59cc843cd6a0a5d70311e70e95b589100ea38302eb7"
Nov 01 20:55:37.929221 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:37.929189    5041 scope.go:122] "RemoveContainer" containerID="2749cf1251cab2f21ce94803d4287142aba23e7f134a4f161703f6932d00fff5"
Nov 01 20:55:41.402575 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.402540    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"]
Nov 01 20:55:41.409959 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.409929    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jbtrl"]
Nov 01 20:55:41.424090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.424058    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"]
Nov 01 20:55:41.424530 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.424518    5041 cpu_manager.go:414] "containerMap: removing container" podUID="f5b4f20d-0020-4379-9935-54deb8995804" containerName="revoke-keys"
Nov 01 20:55:41.424578 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.424532    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b4f20d-0020-4379-9935-54deb8995804" containerName="revoke-keys"
Nov 01 20:55:41.424615 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.424610    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b4f20d-0020-4379-9935-54deb8995804" containerName="revoke-keys"
Nov 01 20:55:41.429458 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.429438    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.434437 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.434412    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:55:41.434558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.434472    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:55:41.434648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.434609    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:55:41.434648 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.434635    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:55:41.438555 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.438075    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"]
Nov 01 20:55:41.529707 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.529670    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.529910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.529714    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.529910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.529857    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwr2t\" (UniqueName: \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.630936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.630898    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.631128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.630946    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.631128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.630991    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwr2t\" (UniqueName: \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.631625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.631604    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.633304 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.633280    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.639182 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.639160    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwr2t\" (UniqueName: \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\") pod \"maas-api-revoke-sub-3d4156f1-wkbqm\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:41.746032 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:41.745957    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:42.019589 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:42.019559    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"]
Nov 01 20:55:42.021117 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:55:42.021093    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf5bc97_98f6_47bd_a7a3_8ec576a3d239.slice/crio-105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00 WatchSource:0}: Error finding container 105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00: Status 404 returned error can't find the container with id 105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00
Nov 01 20:55:42.086574 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:42.086544    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm" event={"ID":"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239","Type":"ContainerStarted","Data":"105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00"}
Nov 01 20:55:42.427625 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:42.427590    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ff1d91-3476-42ce-b2f7-deed05e91133" path="/var/lib/kubelet/pods/f4ff1d91-3476-42ce-b2f7-deed05e91133/volumes"
Nov 01 20:55:43.093766 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:43.093732    5041 generic.go:356] "Generic (PLEG): container finished" podID="4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" containerID="768c094394b73241166438e906889ec72956a447868965c2a9587c669f414001" exitCode=0
Nov 01 20:55:43.093952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:43.093814    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm" event={"ID":"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239","Type":"ContainerDied","Data":"768c094394b73241166438e906889ec72956a447868965c2a9587c669f414001"}
Nov 01 20:55:44.255101 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.255072    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:44.358327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.358232    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\") pod \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") "
Nov 01 20:55:44.358327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.358317    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\" (UniqueName: \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\") pod \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") "
Nov 01 20:55:44.358565 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.358367    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\") pod \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\" (UID: \"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239\") "
Nov 01 20:55:44.358746 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.358721    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca" pod "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" (UID: "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:55:44.360415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.360394    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca" pod "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" (UID: "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:55:44.360513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.360452    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t" pod "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" (UID: "4bf5bc97-98f6-47bd-a7a3-8ec576a3d239"). InnerVolumeSpecName "kube-api-access-qwr2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:55:44.459502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.459468    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:44.459502 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.459495    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwr2t\" (UniqueName: \"kubernetes.io/projected/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-kube-api-access-qwr2t\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:44.459695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:44.459511    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:55:45.107356 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:45.107327    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"
Nov 01 20:55:45.107545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:45.107330    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm" event={"ID":"4bf5bc97-98f6-47bd-a7a3-8ec576a3d239","Type":"ContainerDied","Data":"105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00"}
Nov 01 20:55:45.107545 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:55:45.107449    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105e6c8cc791d514391306084d03fd8e68899e60cd707dba788e68c01e19bd00"
Nov 01 20:55:58.540621 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:55:58.540582    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:56:15.879415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.879329    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:56:15.882151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.880044    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" containerName="revoke-keys"
Nov 01 20:56:15.882151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.880066    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" containerName="revoke-keys"
Nov 01 20:56:15.882151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.880184    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" containerName="revoke-keys"
Nov 01 20:56:15.883983 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.883961    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.890115 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.890093    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e9e72\"/\"default-dockercfg-j97mr\""
Nov 01 20:56:15.890561 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.890542    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-4e9e72\"/\"simulator-single-main-4e9e72-kserve-self-signed-certs\""
Nov 01 20:56:15.891846 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.891824    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e9e72\"/\"openshift-service-ca.crt\""
Nov 01 20:56:15.892017 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.891904    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-4e9e72\"/\"kube-root-ca.crt\""
Nov 01 20:56:15.892521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.892496    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:56:15.897738 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.897716    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.897863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.897755    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hdx\" (UniqueName: \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.897863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.897772    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.897863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.897812    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.898062 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.898026    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.898116 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.898076    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.998649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998615    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hdx\" (UniqueName: \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.998649 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998653    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998679    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998734    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998755    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999020 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.998818    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.999122    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.999195    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:15.999256 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:15.999235    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:16.000972 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:16.000944    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:16.001243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:16.001224    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:16.005990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:16.005968    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hdx\" (UniqueName: \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\") pod \"simulator-single-main-4e9e72-kserve-cc4795658-lf6mm\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") " pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:16.201513 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:16.201413    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:16.481635 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:16.481602    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:56:16.483640 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:56:16.483606    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81adf452_d635_492c_8cb4_e2b96d8a53df.slice/crio-2d749be4366982e844738e161f0b9f027da07f6e598ff9c57b960093e41e8c17 WatchSource:0}: Error finding container 2d749be4366982e844738e161f0b9f027da07f6e598ff9c57b960093e41e8c17: Status 404 returned error can't find the container with id 2d749be4366982e844738e161f0b9f027da07f6e598ff9c57b960093e41e8c17
Nov 01 20:56:17.346305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:17.346268    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" event={"ID":"81adf452-d635-492c-8cb4-e2b96d8a53df","Type":"ContainerStarted","Data":"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"}
Nov 01 20:56:17.346695 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:17.346317    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" event={"ID":"81adf452-d635-492c-8cb4-e2b96d8a53df","Type":"ContainerStarted","Data":"2d749be4366982e844738e161f0b9f027da07f6e598ff9c57b960093e41e8c17"}
Nov 01 20:56:17.361262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:17.361134    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" podStartSLOduration=2.361118838 podStartE2EDuration="2.361118838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:17.360909464 +0000 UTC m=+3465.081988141" watchObservedRunningTime="2026-11-01 20:56:17.361118838 +0000 UTC m=+3465.082197512"
Nov 01 20:56:18.201709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:18.201671    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:26.201544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:26.201506    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:26.215258 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:26.215227    5041 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:26.411151 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:26.411119    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:56:28.541291 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:56:28.541252    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:56:46.251858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.251825    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"]
Nov 01 20:56:46.255462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.255445    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.258910 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.258887    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:56:46.260053 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.260020    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:56:46.260175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.260090    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:56:46.260283 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.260262    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:56:46.263690 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.263668    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"]
Nov 01 20:56:46.410342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.410307    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.410548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.410352    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbrk\" (UniqueName: \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.410548 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.410393    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.511148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.511052    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.511148 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.511145    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbrk\" (UniqueName: \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.511360 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.511219    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.511952 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.511924    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.513542 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.513516    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.519072 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.519044    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbrk\" (UniqueName: \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\") pod \"maas-api-revoke-sub-ffb079ce-crjcf\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.572267 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.572236    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:46.847339 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:46.847306    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"]
Nov 01 20:56:46.849143 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:56:46.849112    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bb9e28_0415_4c76_b260_882316d18735.slice/crio-5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642 WatchSource:0}: Error finding container 5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642: Status 404 returned error can't find the container with id 5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642
Nov 01 20:56:47.537913 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:47.537872    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf" event={"ID":"88bb9e28-0415-4c76-b260-882316d18735","Type":"ContainerStarted","Data":"b460120dab513c08d0e62c7f55f9469e280489e658584c9f6bc225095fb8975f"}
Nov 01 20:56:47.538295 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:47.537937    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf" event={"ID":"88bb9e28-0415-4c76-b260-882316d18735","Type":"ContainerStarted","Data":"5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642"}
Nov 01 20:56:47.553876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:47.553824    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf" podStartSLOduration=1.553809381 podStartE2EDuration="1.553809381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:47.552297116 +0000 UTC m=+3495.273375792" watchObservedRunningTime="2026-11-01 20:56:47.553809381 +0000 UTC m=+3495.274888091"
Nov 01 20:56:48.545482 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:48.545444    5041 generic.go:356] "Generic (PLEG): container finished" podID="88bb9e28-0415-4c76-b260-882316d18735" containerID="b460120dab513c08d0e62c7f55f9469e280489e658584c9f6bc225095fb8975f" exitCode=0
Nov 01 20:56:48.545918 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:48.545501    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf" event={"ID":"88bb9e28-0415-4c76-b260-882316d18735","Type":"ContainerDied","Data":"b460120dab513c08d0e62c7f55f9469e280489e658584c9f6bc225095fb8975f"}
Nov 01 20:56:49.685340 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.685311    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:49.848362 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.848276    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\") pod \"88bb9e28-0415-4c76-b260-882316d18735\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") "
Nov 01 20:56:49.848558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.848420    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\" (UniqueName: \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\") pod \"88bb9e28-0415-4c76-b260-882316d18735\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") "
Nov 01 20:56:49.848558 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.848484    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\") pod \"88bb9e28-0415-4c76-b260-882316d18735\" (UID: \"88bb9e28-0415-4c76-b260-882316d18735\") "
Nov 01 20:56:49.848909 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.848878    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca" pod "88bb9e28-0415-4c76-b260-882316d18735" (UID: "88bb9e28-0415-4c76-b260-882316d18735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:56:49.851165 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.851143    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk" pod "88bb9e28-0415-4c76-b260-882316d18735" (UID: "88bb9e28-0415-4c76-b260-882316d18735"). InnerVolumeSpecName "kube-api-access-kfbrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:56:49.851281 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.851162    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca" pod "88bb9e28-0415-4c76-b260-882316d18735" (UID: "88bb9e28-0415-4c76-b260-882316d18735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:56:49.949440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.949399    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bb9e28-0415-4c76-b260-882316d18735-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:56:49.949440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.949434    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bb9e28-0415-4c76-b260-882316d18735-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:56:49.949440 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:49.949445    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbrk\" (UniqueName: \"kubernetes.io/projected/88bb9e28-0415-4c76-b260-882316d18735-kube-api-access-kfbrk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:56:50.559317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:50.559288    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"
Nov 01 20:56:50.559317 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:50.559309    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf" event={"ID":"88bb9e28-0415-4c76-b260-882316d18735","Type":"ContainerDied","Data":"5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642"}
Nov 01 20:56:50.559543 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:56:50.559354    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba24a55d3b47487f61014cd834b70ea6be8b2f91f6553627b116911fc414642"
Nov 01 20:56:58.542963 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:56:58.542919    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:57:02.095331 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:02.095295    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:57:17.116250 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.116185    5041 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" podUID="81adf452-d635-492c-8cb4-e2b96d8a53df" containerName="main" containerID="cri-o://98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19" gracePeriod=45
Nov 01 20:57:17.366213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.366188    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:57:17.429000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.428917    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\" (UniqueName: \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429000 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.428963    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429033    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\" (UniqueName: \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429080    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429105    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429168    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\") pod \"81adf452-d635-492c-8cb4-e2b96d8a53df\" (UID: \"81adf452-d635-492c-8cb4-e2b96d8a53df\") "
Nov 01 20:57:17.429252 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429233    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:57:17.429609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429450    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:57:17.429609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429516    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:57:17.429609 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.429533    5041 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-tmp-dir\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.431479 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.431450    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "kube-api-access-99hdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:17.431621 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.431570    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:17.431690 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.431663    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm" pod "81adf452-d635-492c-8cb4-e2b96d8a53df" (UID: "81adf452-d635-492c-8cb4-e2b96d8a53df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 20:57:17.530843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.530761    5041 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-dshm\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.530843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.530819    5041 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-model-cache\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.530843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.530834    5041 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81adf452-d635-492c-8cb4-e2b96d8a53df-home\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.530843 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.530849    5041 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81adf452-d635-492c-8cb4-e2b96d8a53df-tls-certs\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.531140 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.530864    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hdx\" (UniqueName: \"kubernetes.io/projected/81adf452-d635-492c-8cb4-e2b96d8a53df-kube-api-access-99hdx\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:17.720167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.720075    5041 generic.go:356] "Generic (PLEG): container finished" podID="81adf452-d635-492c-8cb4-e2b96d8a53df" containerID="98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19" exitCode=0
Nov 01 20:57:17.720167 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.720122    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"
Nov 01 20:57:17.720408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.720157    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" event={"ID":"81adf452-d635-492c-8cb4-e2b96d8a53df","Type":"ContainerDied","Data":"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"}
Nov 01 20:57:17.720408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.720210    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm" event={"ID":"81adf452-d635-492c-8cb4-e2b96d8a53df","Type":"ContainerDied","Data":"2d749be4366982e844738e161f0b9f027da07f6e598ff9c57b960093e41e8c17"}
Nov 01 20:57:17.720408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.720232    5041 scope.go:122] "RemoveContainer" containerID="98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"
Nov 01 20:57:17.730810 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.730791    5041 scope.go:122] "RemoveContainer" containerID="98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"
Nov 01 20:57:17.731088 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:57:17.731066    5041 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19\": container with ID starting with 98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19 not found: ID does not exist" containerID="98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"
Nov 01 20:57:17.731150 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.731098    5041 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19"} err="failed to get container status \"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19\": rpc error: code = NotFound desc = could not find container \"98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19\": container with ID starting with 98896c743acbb237f405090bc399148ac47b0c2ea5f51e941f04b1405fe9db19 not found: ID does not exist"
Nov 01 20:57:17.744819 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.744790    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:57:17.747447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:17.747430    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-4e9e72/simulator-single-main-4e9e72-kserve-cc4795658-lf6mm"]
Nov 01 20:57:18.427630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:18.427588    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81adf452-d635-492c-8cb4-e2b96d8a53df" path="/var/lib/kubelet/pods/81adf452-d635-492c-8cb4-e2b96d8a53df/volumes"
Nov 01 20:57:21.019412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:21.019377    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"]
Nov 01 20:57:21.021892 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:21.021868    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wl55v"]
Nov 01 20:57:22.426940 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:22.426904    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5971f8-5685-44db-86d0-5f27781ead0d" path="/var/lib/kubelet/pods/8a5971f8-5685-44db-86d0-5f27781ead0d/volumes"
Nov 01 20:57:28.540982 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:57:28.540940    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:57:40.021444 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:40.021411    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"]
Nov 01 20:57:40.023863 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:40.023833    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-wgfsf"]
Nov 01 20:57:40.426982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:40.426948    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60888dd8-e975-4d72-bddd-f5b283612a81" path="/var/lib/kubelet/pods/60888dd8-e975-4d72-bddd-f5b283612a81/volumes"
Nov 01 20:57:44.196823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.196724    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"]
Nov 01 20:57:44.197400 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197379    5041 cpu_manager.go:414] "containerMap: removing container" podUID="81adf452-d635-492c-8cb4-e2b96d8a53df" containerName="main"
Nov 01 20:57:44.197492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197403    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81adf452-d635-492c-8cb4-e2b96d8a53df" containerName="main"
Nov 01 20:57:44.197492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197437    5041 cpu_manager.go:414] "containerMap: removing container" podUID="88bb9e28-0415-4c76-b260-882316d18735" containerName="revoke-keys"
Nov 01 20:57:44.197492 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197445    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bb9e28-0415-4c76-b260-882316d18735" containerName="revoke-keys"
Nov 01 20:57:44.197655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197583    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bb9e28-0415-4c76-b260-882316d18735" containerName="revoke-keys"
Nov 01 20:57:44.197655 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.197605    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="81adf452-d635-492c-8cb4-e2b96d8a53df" containerName="main"
Nov 01 20:57:44.202289 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.202266    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.206104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.206077    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:57:44.207153 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.207132    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:57:44.207448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.207425    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:57:44.207809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.207765    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"]
Nov 01 20:57:44.208002 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.207981    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:57:44.291729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.291692    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksbk\" (UniqueName: \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.291945 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.291774    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.291997 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.291970    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.393001 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.392969    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.393180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.393010    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksbk\" (UniqueName: \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.393180 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.393064    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.393709 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.393683    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.395394 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.395371    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.400868 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.400849    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksbk\" (UniqueName: \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.518175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.518089    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:44.797279 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:57:44.797248    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7947c91e_1bc9_473e_a162_2daf09a078ed.slice/crio-52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c WatchSource:0}: Error finding container 52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c: Status 404 returned error can't find the container with id 52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c
Nov 01 20:57:44.797411 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.797310    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"]
Nov 01 20:57:44.888771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:44.888736    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx" event={"ID":"7947c91e-1bc9-473e-a162-2daf09a078ed","Type":"ContainerStarted","Data":"52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c"}
Nov 01 20:57:45.900175 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:45.900091    5041 generic.go:356] "Generic (PLEG): container finished" podID="7947c91e-1bc9-473e-a162-2daf09a078ed" containerID="75ec105463c6e1e805d0c495d5e542b7ffdb44e88dcdbe9a72c087bc0b1cd48e" exitCode=0
Nov 01 20:57:45.900551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:45.900180    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx" event={"ID":"7947c91e-1bc9-473e-a162-2daf09a078ed","Type":"ContainerDied","Data":"75ec105463c6e1e805d0c495d5e542b7ffdb44e88dcdbe9a72c087bc0b1cd48e"}
Nov 01 20:57:47.052531 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.052504    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:47.120391 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.120357    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\" (UniqueName: \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\") pod \"7947c91e-1bc9-473e-a162-2daf09a078ed\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") "
Nov 01 20:57:47.120591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.120415    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\") pod \"7947c91e-1bc9-473e-a162-2daf09a078ed\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") "
Nov 01 20:57:47.120591 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.120522    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\") pod \"7947c91e-1bc9-473e-a162-2daf09a078ed\" (UID: \"7947c91e-1bc9-473e-a162-2daf09a078ed\") "
Nov 01 20:57:47.120947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.120907    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca" pod "7947c91e-1bc9-473e-a162-2daf09a078ed" (UID: "7947c91e-1bc9-473e-a162-2daf09a078ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:47.122698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.122673    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca" pod "7947c91e-1bc9-473e-a162-2daf09a078ed" (UID: "7947c91e-1bc9-473e-a162-2daf09a078ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:47.122823 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.122710    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk" pod "7947c91e-1bc9-473e-a162-2daf09a078ed" (UID: "7947c91e-1bc9-473e-a162-2daf09a078ed"). InnerVolumeSpecName "kube-api-access-pksbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:47.221806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.221691    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:47.221806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.221725    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksbk\" (UniqueName: \"kubernetes.io/projected/7947c91e-1bc9-473e-a162-2daf09a078ed-kube-api-access-pksbk\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:47.221806 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.221735    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7947c91e-1bc9-473e-a162-2daf09a078ed-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:47.913177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.913142    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx" event={"ID":"7947c91e-1bc9-473e-a162-2daf09a078ed","Type":"ContainerDied","Data":"52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c"}
Nov 01 20:57:47.913177 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.913154    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"
Nov 01 20:57:47.913408 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:47.913195    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c757968e0664de286e7746149bfc5215ba3b6124b06f0216ae3c588137f13c"
Nov 01 20:57:51.023327 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:51.023288    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"]
Nov 01 20:57:51.026570 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:51.026545    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-xmtxd"]
Nov 01 20:57:52.427342 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:52.427307    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d19fcb-d894-461c-851d-27838ecfbb4e" path="/var/lib/kubelet/pods/97d19fcb-d894-461c-851d-27838ecfbb4e/volumes"
Nov 01 20:57:54.716771 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:54.716734    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"]
Nov 01 20:57:54.719234 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:54.719212    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4e9e72-vqvrx"]
Nov 01 20:57:56.236022 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.235986    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"]
Nov 01 20:57:56.236433 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.236037    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wkbqm"]
Nov 01 20:57:56.259656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.259625    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"]
Nov 01 20:57:56.260088 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.260075    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7947c91e-1bc9-473e-a162-2daf09a078ed" containerName="revoke-keys"
Nov 01 20:57:56.260152 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.260089    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7947c91e-1bc9-473e-a162-2daf09a078ed" containerName="revoke-keys"
Nov 01 20:57:56.260197 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.260186    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7947c91e-1bc9-473e-a162-2daf09a078ed" containerName="revoke-keys"
Nov 01 20:57:56.263818 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.263772    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.267193 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.267171    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:57:56.267334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.267270    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:57:56.267334 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.267270    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:57:56.268111 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.268090    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:57:56.270240 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.270215    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"]
Nov 01 20:57:56.417659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.417624    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.417864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.417756    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.417864 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.417801    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwvh\" (UniqueName: \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.426404 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.426372    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf5bc97-98f6-47bd-a7a3-8ec576a3d239" path="/var/lib/kubelet/pods/4bf5bc97-98f6-47bd-a7a3-8ec576a3d239/volumes"
Nov 01 20:57:56.426812 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.426798    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7947c91e-1bc9-473e-a162-2daf09a078ed" path="/var/lib/kubelet/pods/7947c91e-1bc9-473e-a162-2daf09a078ed/volumes"
Nov 01 20:57:56.519091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.519007    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.519091 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.519040    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwvh\" (UniqueName: \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.519307 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.519105    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.519751 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.519727    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.521528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.521506    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.527308 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.527283    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwvh\" (UniqueName: \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\") pod \"maas-api-revoke-sub-3d4156f1-p4tg8\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.583247 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.583204    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:56.861645 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.861617    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"]
Nov 01 20:57:56.863771 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:57:56.863742    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84339091_243a_46a3_ab24_df9a957dfd64.slice/crio-b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672 WatchSource:0}: Error finding container b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672: Status 404 returned error can't find the container with id b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672
Nov 01 20:57:56.970659 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:56.970621    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8" event={"ID":"84339091-243a-46a3-ab24-df9a957dfd64","Type":"ContainerStarted","Data":"b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672"}
Nov 01 20:57:57.978051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:57.977958    5041 generic.go:356] "Generic (PLEG): container finished" podID="84339091-243a-46a3-ab24-df9a957dfd64" containerID="3e6ac01277c6e003bad2620504e0ee08bbb66e5951b1b20a8201489ba4e51019" exitCode=0
Nov 01 20:57:57.978051 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:57.978004    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8" event={"ID":"84339091-243a-46a3-ab24-df9a957dfd64","Type":"ContainerDied","Data":"3e6ac01277c6e003bad2620504e0ee08bbb66e5951b1b20a8201489ba4e51019"}
Nov 01 20:57:58.541857 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:57:58.541820    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:57:59.124306 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.124279    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:59.245251 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.245167    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\" (UniqueName: \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\") pod \"84339091-243a-46a3-ab24-df9a957dfd64\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") "
Nov 01 20:57:59.245412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.245291    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\") pod \"84339091-243a-46a3-ab24-df9a957dfd64\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") "
Nov 01 20:57:59.245412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.245366    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\") pod \"84339091-243a-46a3-ab24-df9a957dfd64\" (UID: \"84339091-243a-46a3-ab24-df9a957dfd64\") "
Nov 01 20:57:59.245727 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.245704    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca" pod "84339091-243a-46a3-ab24-df9a957dfd64" (UID: "84339091-243a-46a3-ab24-df9a957dfd64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:57:59.247314 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.247283    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh" pod "84339091-243a-46a3-ab24-df9a957dfd64" (UID: "84339091-243a-46a3-ab24-df9a957dfd64"). InnerVolumeSpecName "kube-api-access-9dwvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:57:59.247425 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.247384    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca" pod "84339091-243a-46a3-ab24-df9a957dfd64" (UID: "84339091-243a-46a3-ab24-df9a957dfd64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:57:59.346501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.346467    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:59.346501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.346495    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84339091-243a-46a3-ab24-df9a957dfd64-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:59.346501 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.346505    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwvh\" (UniqueName: \"kubernetes.io/projected/84339091-243a-46a3-ab24-df9a957dfd64-kube-api-access-9dwvh\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:57:59.991357 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.991323    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8"
Nov 01 20:57:59.991630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.991333    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4tg8" event={"ID":"84339091-243a-46a3-ab24-df9a957dfd64","Type":"ContainerDied","Data":"b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672"}
Nov 01 20:57:59.991630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:57:59.991447    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b762a5f89e547a06760a7599b3ac3a69727e7b19bfffaf7c96e81eef124fa672"
Nov 01 20:58:15.024104 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:15.024067    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"]
Nov 01 20:58:15.027124 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:15.027092    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-2khwh"]
Nov 01 20:58:16.426693 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.426640    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c886f375-21ec-4057-9eb2-32e3fa43d3d9" path="/var/lib/kubelet/pods/c886f375-21ec-4057-9eb2-32e3fa43d3d9/volumes"
Nov 01 20:58:16.745014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.744938    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"]
Nov 01 20:58:16.745399 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.745386    5041 cpu_manager.go:414] "containerMap: removing container" podUID="84339091-243a-46a3-ab24-df9a957dfd64" containerName="revoke-keys"
Nov 01 20:58:16.745447 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.745401    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84339091-243a-46a3-ab24-df9a957dfd64" containerName="revoke-keys"
Nov 01 20:58:16.745485 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.745479    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="84339091-243a-46a3-ab24-df9a957dfd64" containerName="revoke-keys"
Nov 01 20:58:16.748711 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.748695    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.753472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.753447    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:58:16.753472 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.753465    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:58:16.753691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.753475    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:58:16.753691 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.753584    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:58:16.757475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.757452    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"]
Nov 01 20:58:16.848412 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.848378    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.848610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.848434    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.848610 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.848453    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fsnw\" (UniqueName: \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.949448 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.949417    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.949642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.949469    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.949642 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.949594    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsnw\" (UniqueName: \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.950099 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.950075    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.951917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.951891    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:16.957190 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:16.957166    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fsnw\" (UniqueName: \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\") pod \"maas-api-revoke-sub-d2bc93bc-h4562\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:17.063938 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:17.063904    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:17.337870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:17.337844    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"]
Nov 01 20:58:17.343134 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:58:17.343091    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed93b50_80aa_488a_ad06_fe93447b6065.slice/crio-dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07 WatchSource:0}: Error finding container dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07: Status 404 returned error can't find the container with id dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07
Nov 01 20:58:18.145672 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:18.145638    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562" event={"ID":"8ed93b50-80aa-488a-ad06-fe93447b6065","Type":"ContainerStarted","Data":"b6662d79e9bf578da6f0ba858fed06103faa055b8fc411b31965d5d2c312b466"}
Nov 01 20:58:18.146142 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:18.145691    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562" event={"ID":"8ed93b50-80aa-488a-ad06-fe93447b6065","Type":"ContainerStarted","Data":"dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07"}
Nov 01 20:58:18.161489 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:18.161427    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562" podStartSLOduration=2.161407461 podStartE2EDuration="2.161407461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:18.159913417 +0000 UTC m=+3585.880992093" watchObservedRunningTime="2026-11-01 20:58:18.161407461 +0000 UTC m=+3585.882486138"
Nov 01 20:58:19.153461 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:19.153425    5041 generic.go:356] "Generic (PLEG): container finished" podID="8ed93b50-80aa-488a-ad06-fe93447b6065" containerID="b6662d79e9bf578da6f0ba858fed06103faa055b8fc411b31965d5d2c312b466" exitCode=0
Nov 01 20:58:19.153934 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:19.153507    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562" event={"ID":"8ed93b50-80aa-488a-ad06-fe93447b6065","Type":"ContainerDied","Data":"b6662d79e9bf578da6f0ba858fed06103faa055b8fc411b31965d5d2c312b466"}
Nov 01 20:58:20.323793 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.323753    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:20.381835 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.381806    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\") pod \"8ed93b50-80aa-488a-ad06-fe93447b6065\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") "
Nov 01 20:58:20.382014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.381844    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\") pod \"8ed93b50-80aa-488a-ad06-fe93447b6065\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") "
Nov 01 20:58:20.382014 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.381876    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\" (UniqueName: \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\") pod \"8ed93b50-80aa-488a-ad06-fe93447b6065\" (UID: \"8ed93b50-80aa-488a-ad06-fe93447b6065\") "
Nov 01 20:58:20.382312 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.382274    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca" pod "8ed93b50-80aa-488a-ad06-fe93447b6065" (UID: "8ed93b50-80aa-488a-ad06-fe93447b6065"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:20.384064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.384040    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca" pod "8ed93b50-80aa-488a-ad06-fe93447b6065" (UID: "8ed93b50-80aa-488a-ad06-fe93447b6065"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:58:20.384382 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.384362    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw" pod "8ed93b50-80aa-488a-ad06-fe93447b6065" (UID: "8ed93b50-80aa-488a-ad06-fe93447b6065"). InnerVolumeSpecName "kube-api-access-4fsnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:20.482876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.482844    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:20.482876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.482871    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ed93b50-80aa-488a-ad06-fe93447b6065-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:20.482876 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:20.482880    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fsnw\" (UniqueName: \"kubernetes.io/projected/8ed93b50-80aa-488a-ad06-fe93447b6065-kube-api-access-4fsnw\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:21.166445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:21.166412    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562"
Nov 01 20:58:21.166445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:21.166434    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-h4562" event={"ID":"8ed93b50-80aa-488a-ad06-fe93447b6065","Type":"ContainerDied","Data":"dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07"}
Nov 01 20:58:21.166674 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:21.166508    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde9a11b36accd64f62fb31ed461a8d1d56f531904a5fba0be4036069c601b07"
Nov 01 20:58:25.020380 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:25.020342    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"]
Nov 01 20:58:25.023683 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:25.023654    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b99cn"]
Nov 01 20:58:26.427262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:26.427228    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d6dbcc-ab32-40d2-a9a4-198113213d30" path="/var/lib/kubelet/pods/f1d6dbcc-ab32-40d2-a9a4-198113213d30/volumes"
Nov 01 20:58:28.541760 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:58:28.541722    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:58:33.163126 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:33.163095    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:58:33.181323 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:33.181299    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 20:58:37.988838 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:37.988803    5041 scope.go:122] "RemoveContainer" containerID="5464ba4976fea485ef0fbb6f14841d33d6862b93e861ebc0668fabc710027322"
Nov 01 20:58:38.013471 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:38.013150    5041 scope.go:122] "RemoveContainer" containerID="aa993b2acb6be0f56372b7feebeecf89b128178f1f33ea46fa938ae038913576"
Nov 01 20:58:45.021816 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.021767    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"]
Nov 01 20:58:45.024168 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.024144    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-j74rj"]
Nov 01 20:58:45.542935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.542900    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"]
Nov 01 20:58:45.543429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.543415    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8ed93b50-80aa-488a-ad06-fe93447b6065" containerName="revoke-keys"
Nov 01 20:58:45.543474 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.543430    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed93b50-80aa-488a-ad06-fe93447b6065" containerName="revoke-keys"
Nov 01 20:58:45.543521 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.543512    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed93b50-80aa-488a-ad06-fe93447b6065" containerName="revoke-keys"
Nov 01 20:58:45.548305 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.548283    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.552128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.552055    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:58:45.552128 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.552102    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:58:45.552608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.552530    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:58:45.552608 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.552575    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:58:45.554760 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.554731    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"]
Nov 01 20:58:45.649885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.649850    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.649885 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.649886    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfrr\" (UniqueName: \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.650098 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.650012    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.750837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.750766    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.750837 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.750840    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfrr\" (UniqueName: \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.751076 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.750883    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.751413 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.751390    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.753272 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.753254    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.758547 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.758525    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfrr\" (UniqueName: \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\") pod \"maas-api-revoke-sub-84c616b6-rgvsd\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:45.865653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:45.865577    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:46.140415 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:46.140389    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"]
Nov 01 20:58:46.142140 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:58:46.142112    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0f9bda_5cee_4500_a0db_3f6698821391.slice/crio-96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4 WatchSource:0}: Error finding container 96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4: Status 404 returned error can't find the container with id 96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4
Nov 01 20:58:46.363581 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:46.363545    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd" event={"ID":"7d0f9bda-5cee-4500-a0db-3f6698821391","Type":"ContainerStarted","Data":"96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4"}
Nov 01 20:58:46.427365 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:46.427287    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1883233-5c9d-40ca-8b47-8b64efe7ccfd" path="/var/lib/kubelet/pods/d1883233-5c9d-40ca-8b47-8b64efe7ccfd/volumes"
Nov 01 20:58:47.371010 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:47.370973    5041 generic.go:356] "Generic (PLEG): container finished" podID="7d0f9bda-5cee-4500-a0db-3f6698821391" containerID="a4aafbf7c03af9fd88e3259fcfc0f41c56ff575fd19b28b5caf54eda08cc122b" exitCode=0
Nov 01 20:58:47.371494 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:47.371021    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd" event={"ID":"7d0f9bda-5cee-4500-a0db-3f6698821391","Type":"ContainerDied","Data":"a4aafbf7c03af9fd88e3259fcfc0f41c56ff575fd19b28b5caf54eda08cc122b"}
Nov 01 20:58:48.515943 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.515917    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:48.580083 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.580039    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\") pod \"7d0f9bda-5cee-4500-a0db-3f6698821391\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") "
Nov 01 20:58:48.580296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.580120    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\" (UniqueName: \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\") pod \"7d0f9bda-5cee-4500-a0db-3f6698821391\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") "
Nov 01 20:58:48.580296 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.580168    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\") pod \"7d0f9bda-5cee-4500-a0db-3f6698821391\" (UID: \"7d0f9bda-5cee-4500-a0db-3f6698821391\") "
Nov 01 20:58:48.580569 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.580546    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca" pod "7d0f9bda-5cee-4500-a0db-3f6698821391" (UID: "7d0f9bda-5cee-4500-a0db-3f6698821391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:48.582225 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.582204    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca" pod "7d0f9bda-5cee-4500-a0db-3f6698821391" (UID: "7d0f9bda-5cee-4500-a0db-3f6698821391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:58:48.582338 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.582235    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr" pod "7d0f9bda-5cee-4500-a0db-3f6698821391" (UID: "7d0f9bda-5cee-4500-a0db-3f6698821391"). InnerVolumeSpecName "kube-api-access-ptfrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:48.681497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.681407    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptfrr\" (UniqueName: \"kubernetes.io/projected/7d0f9bda-5cee-4500-a0db-3f6698821391-kube-api-access-ptfrr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:48.681497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.681441    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:48.681497 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:48.681456    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d0f9bda-5cee-4500-a0db-3f6698821391-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:49.384729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:49.384697    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd"
Nov 01 20:58:49.384729 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:49.384721    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84c616b6-rgvsd" event={"ID":"7d0f9bda-5cee-4500-a0db-3f6698821391","Type":"ContainerDied","Data":"96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4"}
Nov 01 20:58:49.384968 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:49.384799    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96052c4daf89569c306733094d9bbc24ebc570e843fed2cdb9bb19977a2a0aa4"
Nov 01 20:58:55.755090 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.755054    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"]
Nov 01 20:58:55.755722 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.755699    5041 cpu_manager.go:414] "containerMap: removing container" podUID="7d0f9bda-5cee-4500-a0db-3f6698821391" containerName="revoke-keys"
Nov 01 20:58:55.755857 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.755724    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0f9bda-5cee-4500-a0db-3f6698821391" containerName="revoke-keys"
Nov 01 20:58:55.755926 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.755898    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0f9bda-5cee-4500-a0db-3f6698821391" containerName="revoke-keys"
Nov 01 20:58:55.759557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.759536    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.766795 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.766082    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:58:55.771265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.767888    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:58:55.771265 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.768077    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:58:55.772809 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.772770    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:58:55.773748 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.773718    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"]
Nov 01 20:58:55.854245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.854213    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.854442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.854273    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnj8\" (UniqueName: \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.854442 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.854407    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.955429 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.955385    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.955630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.955452    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnj8\" (UniqueName: \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.955630 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.955496    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.956191 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.956163    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.957917 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.957895    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:55.963057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:55.963033    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnj8\" (UniqueName: \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\") pod \"maas-api-revoke-sub-eefe487b-f5dg5\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:56.087087 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:56.087050    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:56.365811 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:56.365762    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"]
Nov 01 20:58:56.367553 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:58:56.367520    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd92a18_1af7_4bc8_adbe_579f02c5c8c4.slice/crio-04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda WatchSource:0}: Error finding container 04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda: Status 404 returned error can't find the container with id 04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda
Nov 01 20:58:56.428557 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:56.428527    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5" event={"ID":"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4","Type":"ContainerStarted","Data":"04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda"}
Nov 01 20:58:57.436350 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:57.436262    5041 generic.go:356] "Generic (PLEG): container finished" podID="4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" containerID="1f438b19fac6a9e82ccbe3b7f217a40393d3f6be465fea9e607d5b80656432ef" exitCode=0
Nov 01 20:58:57.436720 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:57.436353    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5" event={"ID":"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4","Type":"ContainerDied","Data":"1f438b19fac6a9e82ccbe3b7f217a40393d3f6be465fea9e607d5b80656432ef"}
Nov 01 20:58:58.544637 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:58:58.544599    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:58:58.584841 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.584811    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:58.683058 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.683021    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\") pod \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") "
Nov 01 20:58:58.683239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.683157    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\") pod \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") "
Nov 01 20:58:58.683239 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.683186    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\" (UniqueName: \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\") pod \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\" (UID: \"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4\") "
Nov 01 20:58:58.683551 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.683526    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca" pod "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" (UID: "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:58:58.685209 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.685183    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca" pod "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" (UID: "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:58:58.685302 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.685270    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8" pod "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" (UID: "4fd92a18-1af7-4bc8-adbe-579f02c5c8c4"). InnerVolumeSpecName "kube-api-access-vmnj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:58:58.784224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.784142    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:58.784224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.784173    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnj8\" (UniqueName: \"kubernetes.io/projected/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-kube-api-access-vmnj8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:58.784224 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:58.784188    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd92a18-1af7-4bc8-adbe-579f02c5c8c4-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:58:59.450477 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:59.450448    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5"
Nov 01 20:58:59.450666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:59.450478    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eefe487b-f5dg5" event={"ID":"4fd92a18-1af7-4bc8-adbe-579f02c5c8c4","Type":"ContainerDied","Data":"04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda"}
Nov 01 20:58:59.450666 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:58:59.450528    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04db2a44b859ddea702d696ceec84364a758fc05db503491f1018b87b491adda"
Nov 01 20:59:01.021858 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:01.021818    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"]
Nov 01 20:59:01.024499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:01.024469    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-gfqwf"]
Nov 01 20:59:02.427529 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:02.427495    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db9ad78-79f5-4b21-a383-ada349450d85" path="/var/lib/kubelet/pods/3db9ad78-79f5-4b21-a383-ada349450d85/volumes"
Nov 01 20:59:15.382285 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.382186    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"]
Nov 01 20:59:15.382860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.382683    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" containerName="revoke-keys"
Nov 01 20:59:15.382860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.382698    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" containerName="revoke-keys"
Nov 01 20:59:15.382860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.382824    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd92a18-1af7-4bc8-adbe-579f02c5c8c4" containerName="revoke-keys"
Nov 01 20:59:15.386204 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.386188    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.390573 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.390543    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:59:15.390723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.390596    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:59:15.390723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.390656    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:59:15.390723 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.390716    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:59:15.392953 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.392927    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"]
Nov 01 20:59:15.457883 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.457841    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.458048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.457970    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sjvr\" (UniqueName: \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.458179 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.458157    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.559602 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.559571    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.559821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.559654    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.559821 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.559718    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sjvr\" (UniqueName: \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.560428 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.560402    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.562008 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.561976    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.567487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.567466    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sjvr\" (UniqueName: \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\") pod \"maas-api-revoke-sub-d8d0d18c-pp7w8\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.702663 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.702569    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:15.975867 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:15.975837    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"]
Nov 01 20:59:15.977960 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:59:15.977934    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424832e3_d49c_4772_a266_d6f7d3a40f5b.slice/crio-48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6 WatchSource:0}: Error finding container 48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6: Status 404 returned error can't find the container with id 48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6
Nov 01 20:59:16.020245 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.020217    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"]
Nov 01 20:59:16.024345 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.024324    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7shzh"]
Nov 01 20:59:16.427195 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.427160    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f076a401-64f0-4832-aa5f-bb3174eb1807" path="/var/lib/kubelet/pods/f076a401-64f0-4832-aa5f-bb3174eb1807/volumes"
Nov 01 20:59:16.556982 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.556950    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8" event={"ID":"424832e3-d49c-4772-a266-d6f7d3a40f5b","Type":"ContainerStarted","Data":"32bc95a8f60225af5eb6869858ff4351f3f64aa8b66653667fdde4e76aad61ba"}
Nov 01 20:59:16.557196 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.557005    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8" event={"ID":"424832e3-d49c-4772-a266-d6f7d3a40f5b","Type":"ContainerStarted","Data":"48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6"}
Nov 01 20:59:16.574311 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:16.574253    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8" podStartSLOduration=1.574235684 podStartE2EDuration="1.574235684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:59:16.571969377 +0000 UTC m=+3644.293048043" watchObservedRunningTime="2026-11-01 20:59:16.574235684 +0000 UTC m=+3644.295314361"
Nov 01 20:59:17.564299 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:17.564256    5041 generic.go:356] "Generic (PLEG): container finished" podID="424832e3-d49c-4772-a266-d6f7d3a40f5b" containerID="32bc95a8f60225af5eb6869858ff4351f3f64aa8b66653667fdde4e76aad61ba" exitCode=0
Nov 01 20:59:17.564698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:17.564313    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8" event={"ID":"424832e3-d49c-4772-a266-d6f7d3a40f5b","Type":"ContainerDied","Data":"32bc95a8f60225af5eb6869858ff4351f3f64aa8b66653667fdde4e76aad61ba"}
Nov 01 20:59:18.707089 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.707062    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:18.794243 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.794201    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\" (UniqueName: \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\") pod \"424832e3-d49c-4772-a266-d6f7d3a40f5b\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") "
Nov 01 20:59:18.794446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.794354    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\") pod \"424832e3-d49c-4772-a266-d6f7d3a40f5b\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") "
Nov 01 20:59:18.794446 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.794406    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\") pod \"424832e3-d49c-4772-a266-d6f7d3a40f5b\" (UID: \"424832e3-d49c-4772-a266-d6f7d3a40f5b\") "
Nov 01 20:59:18.794824 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.794796    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca" pod "424832e3-d49c-4772-a266-d6f7d3a40f5b" (UID: "424832e3-d49c-4772-a266-d6f7d3a40f5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:59:18.796421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.796397    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr" pod "424832e3-d49c-4772-a266-d6f7d3a40f5b" (UID: "424832e3-d49c-4772-a266-d6f7d3a40f5b"). InnerVolumeSpecName "kube-api-access-4sjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:59:18.796421 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.796398    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca" pod "424832e3-d49c-4772-a266-d6f7d3a40f5b" (UID: "424832e3-d49c-4772-a266-d6f7d3a40f5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:59:18.895271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.895197    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sjvr\" (UniqueName: \"kubernetes.io/projected/424832e3-d49c-4772-a266-d6f7d3a40f5b-kube-api-access-4sjvr\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:18.895271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.895226    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:18.895271 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:18.895236    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424832e3-d49c-4772-a266-d6f7d3a40f5b-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:19.577366 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:19.577338    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8"
Nov 01 20:59:19.577577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:19.577373    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d0d18c-pp7w8" event={"ID":"424832e3-d49c-4772-a266-d6f7d3a40f5b","Type":"ContainerDied","Data":"48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6"}
Nov 01 20:59:19.577577 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:19.577420    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48b435d00cbbcc008f9aeb0fe167be2df730ce41eacca3c3c1ffaae4e5d000b6"
Nov 01 20:59:28.548232 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:59:28.548187    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 20:59:31.020031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:31.019984    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"]
Nov 01 20:59:31.022346 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:31.022318    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-94xfn"]
Nov 01 20:59:32.428678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:32.428643    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196857ee-6fc6-4864-afa2-a18fcaa6c257" path="/var/lib/kubelet/pods/196857ee-6fc6-4864-afa2-a18fcaa6c257/volumes"
Nov 01 20:59:38.105935 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:38.105906    5041 scope.go:122] "RemoveContainer" containerID="0052924790e5d2b7091663bd3de1eb8a92d61c2b927bede08dc0b065a5825d2e"
Nov 01 20:59:38.114977 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:38.114956    5041 scope.go:122] "RemoveContainer" containerID="1062ab28a68af30ae438b84d08e61a4cdd6a152c7f23498332a60bcdc4f5304b"
Nov 01 20:59:38.122989 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:38.122960    5041 scope.go:122] "RemoveContainer" containerID="d11004452919eabb204586ddf3c56aafff72c4724738ea078b0f869146a694f5"
Nov 01 20:59:42.122212 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.122172    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"]
Nov 01 20:59:42.122870 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.122849    5041 cpu_manager.go:414] "containerMap: removing container" podUID="424832e3-d49c-4772-a266-d6f7d3a40f5b" containerName="revoke-keys"
Nov 01 20:59:42.122990 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.122871    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424832e3-d49c-4772-a266-d6f7d3a40f5b" containerName="revoke-keys"
Nov 01 20:59:42.123057 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.123028    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="424832e3-d49c-4772-a266-d6f7d3a40f5b" containerName="revoke-keys"
Nov 01 20:59:42.127942 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.127920    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.131856 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.131833    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:59:42.133048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.133021    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:59:42.133048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.133048    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:59:42.133048 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.133091    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:59:42.133528 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.133317    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"]
Nov 01 20:59:42.213324 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.213293    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.213499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.213354    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrsg\" (UniqueName: \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.213499 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.213469    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.314698 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.314658    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.314936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.314767    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrsg\" (UniqueName: \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.314936 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.314886    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.315631 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.315608    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.317082 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.317060    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.322650 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.322616    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrsg\" (UniqueName: \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\") pod \"maas-api-revoke-sub-32889fd7-7tsnr\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.443337 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.443261    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:42.716544 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.716515    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"]
Nov 01 20:59:42.718147 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:59:42.718119    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275f3e08_3d1b_4a09_bcb4_df9f9e561c83.slice/crio-1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e WatchSource:0}: Error finding container 1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e: Status 404 returned error can't find the container with id 1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e
Nov 01 20:59:42.828656 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:42.828619    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr" event={"ID":"275f3e08-3d1b-4a09-bcb4-df9f9e561c83","Type":"ContainerStarted","Data":"1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e"}
Nov 01 20:59:43.836269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:43.836168    5041 generic.go:356] "Generic (PLEG): container finished" podID="275f3e08-3d1b-4a09-bcb4-df9f9e561c83" containerID="f75856424a4ee072124e8e128fa534a42092d78d43e9e154d05582d53bc0aa5a" exitCode=0
Nov 01 20:59:43.836269 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:43.836250    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr" event={"ID":"275f3e08-3d1b-4a09-bcb4-df9f9e561c83","Type":"ContainerDied","Data":"f75856424a4ee072124e8e128fa534a42092d78d43e9e154d05582d53bc0aa5a"}
Nov 01 20:59:44.994439 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:44.994415    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:45.037487 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.037447    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\") pod \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") "
Nov 01 20:59:45.037678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.037537    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\") pod \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") "
Nov 01 20:59:45.037678 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.037634    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\" (UniqueName: \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\") pod \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\" (UID: \"275f3e08-3d1b-4a09-bcb4-df9f9e561c83\") "
Nov 01 20:59:45.037895 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.037868    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca" pod "275f3e08-3d1b-4a09-bcb4-df9f9e561c83" (UID: "275f3e08-3d1b-4a09-bcb4-df9f9e561c83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:59:45.038042 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.038025    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:45.039927 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.039902    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg" pod "275f3e08-3d1b-4a09-bcb4-df9f9e561c83" (UID: "275f3e08-3d1b-4a09-bcb4-df9f9e561c83"). InnerVolumeSpecName "kube-api-access-lrrsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:59:45.040095 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.040077    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca" pod "275f3e08-3d1b-4a09-bcb4-df9f9e561c83" (UID: "275f3e08-3d1b-4a09-bcb4-df9f9e561c83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:59:45.139432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.139349    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:45.139432 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.139381    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrsg\" (UniqueName: \"kubernetes.io/projected/275f3e08-3d1b-4a09-bcb4-df9f9e561c83-kube-api-access-lrrsg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:45.855554 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.855524    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr"
Nov 01 20:59:45.855772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.855564    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32889fd7-7tsnr" event={"ID":"275f3e08-3d1b-4a09-bcb4-df9f9e561c83","Type":"ContainerDied","Data":"1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e"}
Nov 01 20:59:45.855772 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:45.855627    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1857ace8148a27faddb6df5de14391f9bb2cb914b5d3d3099b8067001844491e"
Nov 01 20:59:52.317860 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.317819    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"]
Nov 01 20:59:52.318462 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.318445    5041 cpu_manager.go:414] "containerMap: removing container" podUID="275f3e08-3d1b-4a09-bcb4-df9f9e561c83" containerName="revoke-keys"
Nov 01 20:59:52.318514 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.318464    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275f3e08-3d1b-4a09-bcb4-df9f9e561c83" containerName="revoke-keys"
Nov 01 20:59:52.318627 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.318616    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="275f3e08-3d1b-4a09-bcb4-df9f9e561c83" containerName="revoke-keys"
Nov 01 20:59:52.322550 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.322521    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.328064 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.328038    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 20:59:52.328262 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.328123    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 20:59:52.334981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.334734    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 20:59:52.335981 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.335959    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 20:59:52.340384 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.340361    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"]
Nov 01 20:59:52.418159 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.418125    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.418341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.418172    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lcx\" (UniqueName: \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.418341 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.418199    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.519475 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.519437    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.519653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.519487    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lcx\" (UniqueName: \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.519653 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.519511    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.520132 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.520105    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.521886 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.521864    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.527468 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.527445    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lcx\" (UniqueName: \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\") pod \"maas-api-revoke-sub-51617705-slmht\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.657939 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.657856    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:52.933122 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:52.933092    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"]
Nov 01 20:59:52.934068 ip-10-0-137-175 kubenswrapper[5041]: W1101 20:59:52.934042    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081ad37a_5755_41ea_8b56_7b40597cd747.slice/crio-b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0 WatchSource:0}: Error finding container b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0: Status 404 returned error can't find the container with id b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0
Nov 01 20:59:53.906987 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:53.906944    5041 generic.go:356] "Generic (PLEG): container finished" podID="081ad37a-5755-41ea-8b56-7b40597cd747" containerID="7535732dd0b83e73b30453b520163965c1f1d2f2d8b16ec5893ae28715ca393b" exitCode=0
Nov 01 20:59:53.907398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:53.907028    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht" event={"ID":"081ad37a-5755-41ea-8b56-7b40597cd747","Type":"ContainerDied","Data":"7535732dd0b83e73b30453b520163965c1f1d2f2d8b16ec5893ae28715ca393b"}
Nov 01 20:59:53.907398 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:53.907088    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht" event={"ID":"081ad37a-5755-41ea-8b56-7b40597cd747","Type":"ContainerStarted","Data":"b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0"}
Nov 01 20:59:55.045246 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.045221    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:55.144947 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.144915    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\") pod \"081ad37a-5755-41ea-8b56-7b40597cd747\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") "
Nov 01 20:59:55.145137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.145002    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\" (UniqueName: \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\") pod \"081ad37a-5755-41ea-8b56-7b40597cd747\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") "
Nov 01 20:59:55.145137 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.145045    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\") pod \"081ad37a-5755-41ea-8b56-7b40597cd747\" (UID: \"081ad37a-5755-41ea-8b56-7b40597cd747\") "
Nov 01 20:59:55.145445 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.145418    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca" pod "081ad37a-5755-41ea-8b56-7b40597cd747" (UID: "081ad37a-5755-41ea-8b56-7b40597cd747"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 20:59:55.147149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.147118    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca" pod "081ad37a-5755-41ea-8b56-7b40597cd747" (UID: "081ad37a-5755-41ea-8b56-7b40597cd747"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 20:59:55.147149 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.147127    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx" pod "081ad37a-5755-41ea-8b56-7b40597cd747" (UID: "081ad37a-5755-41ea-8b56-7b40597cd747"). InnerVolumeSpecName "kube-api-access-g8lcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 20:59:55.246026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.245940    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:55.246026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.245971    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081ad37a-5755-41ea-8b56-7b40597cd747-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:55.246026 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.245981    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8lcx\" (UniqueName: \"kubernetes.io/projected/081ad37a-5755-41ea-8b56-7b40597cd747-kube-api-access-g8lcx\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 20:59:55.921031 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.920998    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht"
Nov 01 20:59:55.921213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.921031    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51617705-slmht" event={"ID":"081ad37a-5755-41ea-8b56-7b40597cd747","Type":"ContainerDied","Data":"b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0"}
Nov 01 20:59:55.921213 ip-10-0-137-175 kubenswrapper[5041]: I1101 20:59:55.921078    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5777944ac47dddb79339c92203e922808a41dd1c99122594c9a0b28700caec0"
Nov 01 20:59:58.542430 ip-10-0-137-175 kubenswrapper[5041]: E1101 20:59:58.542392    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 21:00:00.128503 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.128467    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:00.128973 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.128960    5041 cpu_manager.go:414] "containerMap: removing container" podUID="081ad37a-5755-41ea-8b56-7b40597cd747" containerName="revoke-keys"
Nov 01 21:00:00.129030 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.128974    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081ad37a-5755-41ea-8b56-7b40597cd747" containerName="revoke-keys"
Nov 01 21:00:00.129069 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.129050    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="081ad37a-5755-41ea-8b56-7b40597cd747" containerName="revoke-keys"
Nov 01 21:00:00.132394 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.132375    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.135951 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.135929    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:00:00.136371 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.136344    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:00:00.137991 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.137970    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 21:00:00.139223 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.139203    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:00.197466 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.197430    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbs8r\" (UniqueName: \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.197650 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.197516    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.298926 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.298884    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.299116 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.298985    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbs8r\" (UniqueName: \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.299611 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.299582    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.306541 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.306508    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbs8r\" (UniqueName: \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\") pod \"maas-api-key-cleanup-29892780-6s6jz\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.448195 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.448105    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:00.724757 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.724732    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:00.726280 ip-10-0-137-175 kubenswrapper[5041]: W1101 21:00:00.726245    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dab7faa_1a42_4139_9211_12bb1faf481c.slice/crio-ede05e2095368cec64c0130da2cf22708255b1f531622dd9d2bd2f7f726f49de WatchSource:0}: Error finding container ede05e2095368cec64c0130da2cf22708255b1f531622dd9d2bd2f7f726f49de: Status 404 returned error can't find the container with id ede05e2095368cec64c0130da2cf22708255b1f531622dd9d2bd2f7f726f49de
Nov 01 21:00:00.954472 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:00.954441    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerStarted","Data":"ede05e2095368cec64c0130da2cf22708255b1f531622dd9d2bd2f7f726f49de"}
Nov 01 21:00:01.961661 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:01.961571    5041 generic.go:356] "Generic (PLEG): container finished" podID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerID="197c91183e32b91d4370eb8cdd995607f63a4046567a33ac9ebdedb8bcf48993" exitCode=60
Nov 01 21:00:01.962132 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:01.961652    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerDied","Data":"197c91183e32b91d4370eb8cdd995607f63a4046567a33ac9ebdedb8bcf48993"}
Nov 01 21:00:01.962132 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:01.961910    5041 scope.go:122] "RemoveContainer" containerID="197c91183e32b91d4370eb8cdd995607f63a4046567a33ac9ebdedb8bcf48993"
Nov 01 21:00:02.969844 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:02.969810    5041 generic.go:356] "Generic (PLEG): container finished" podID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerID="a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3" exitCode=60
Nov 01 21:00:02.970343 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:02.969902    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerDied","Data":"a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3"}
Nov 01 21:00:02.970343 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:02.969971    5041 scope.go:122] "RemoveContainer" containerID="197c91183e32b91d4370eb8cdd995607f63a4046567a33ac9ebdedb8bcf48993"
Nov 01 21:00:02.970343 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:02.970203    5041 scope.go:122] "RemoveContainer" containerID="a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3"
Nov 01 21:00:02.970496 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:00:02.970474    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6s6jz_odh-ai-gateway-infra(8dab7faa-1a42-4139-9211-12bb1faf481c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c"
Nov 01 21:00:03.978302 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:03.978270    5041 scope.go:122] "RemoveContainer" containerID="a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3"
Nov 01 21:00:03.978858 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:00:03.978580    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6s6jz_odh-ai-gateway-infra(8dab7faa-1a42-4139-9211-12bb1faf481c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c"
Nov 01 21:00:15.421306 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:15.421268    5041 scope.go:122] "RemoveContainer" containerID="a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3"
Nov 01 21:00:16.098940 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:16.098841    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerStarted","Data":"17eb4991f43e3c93bb0eed102ddb0794db70cbb535070d53c3eb73891640f58e"}
Nov 01 21:00:16.124546 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:16.124486    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" podStartSLOduration=16.124468536 podStartE2EDuration="16.124468536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:16.122758006 +0000 UTC m=+3703.843836682" watchObservedRunningTime="2026-11-01 21:00:16.124468536 +0000 UTC m=+3703.845547213"
Nov 01 21:00:17.121802 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:17.121095    5041 generic.go:356] "Generic (PLEG): container finished" podID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerID="17eb4991f43e3c93bb0eed102ddb0794db70cbb535070d53c3eb73891640f58e" exitCode=60
Nov 01 21:00:17.121802 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:17.121173    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerDied","Data":"17eb4991f43e3c93bb0eed102ddb0794db70cbb535070d53c3eb73891640f58e"}
Nov 01 21:00:17.121802 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:17.121255    5041 scope.go:122] "RemoveContainer" containerID="a3e0c84959066e852a819054ec2c0e46c96897c5a27f561adc3c9f7a14a79ae3"
Nov 01 21:00:17.121802 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:17.121558    5041 scope.go:122] "RemoveContainer" containerID="17eb4991f43e3c93bb0eed102ddb0794db70cbb535070d53c3eb73891640f58e"
Nov 01 21:00:17.122513 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:00:17.121817    5041 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892780-6s6jz_odh-ai-gateway-infra(8dab7faa-1a42-4139-9211-12bb1faf481c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c"
Nov 01 21:00:17.134882 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:17.134858    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:18.269884 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.269858    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:18.349187 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.349153    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\") pod \"8dab7faa-1a42-4139-9211-12bb1faf481c\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") "
Nov 01 21:00:18.349389 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.349202    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\" (UniqueName: \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\") pod \"8dab7faa-1a42-4139-9211-12bb1faf481c\" (UID: \"8dab7faa-1a42-4139-9211-12bb1faf481c\") "
Nov 01 21:00:18.349514 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.349492    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca" pod "8dab7faa-1a42-4139-9211-12bb1faf481c" (UID: "8dab7faa-1a42-4139-9211-12bb1faf481c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:18.349576 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.349567    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dab7faa-1a42-4139-9211-12bb1faf481c-maas-api-service-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:18.351355 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.351326    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r" pod "8dab7faa-1a42-4139-9211-12bb1faf481c" (UID: "8dab7faa-1a42-4139-9211-12bb1faf481c"). InnerVolumeSpecName "kube-api-access-qbs8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:18.450949 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:18.450916    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbs8r\" (UniqueName: \"kubernetes.io/projected/8dab7faa-1a42-4139-9211-12bb1faf481c-kube-api-access-qbs8r\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:19.136337 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:19.136309    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"
Nov 01 21:00:19.136526 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:19.136307    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz" event={"ID":"8dab7faa-1a42-4139-9211-12bb1faf481c","Type":"ContainerDied","Data":"ede05e2095368cec64c0130da2cf22708255b1f531622dd9d2bd2f7f726f49de"}
Nov 01 21:00:19.136526 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:19.136446    5041 scope.go:122] "RemoveContainer" containerID="17eb4991f43e3c93bb0eed102ddb0794db70cbb535070d53c3eb73891640f58e"
Nov 01 21:00:19.159653 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:19.159623    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:19.163510 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:19.163488    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892780-6s6jz"]
Nov 01 21:00:20.031996 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:20.031956    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"]
Nov 01 21:00:20.034356 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:20.034333    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-kjlcb"]
Nov 01 21:00:20.428244 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:20.428204    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f25675d-8378-4940-8175-89ab15524c9b" path="/var/lib/kubelet/pods/3f25675d-8378-4940-8175-89ab15524c9b/volumes"
Nov 01 21:00:20.428653 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:20.428638    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" path="/var/lib/kubelet/pods/8dab7faa-1a42-4139-9211-12bb1faf481c/volumes"
Nov 01 21:00:21.917129 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917085    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"]
Nov 01 21:00:21.917772 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917749    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.917889 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917790    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.917889 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917866    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.917889 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917875    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.918066 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917901    5041 cpu_manager.go:414] "containerMap: removing container" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.918066 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.917910    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.918066 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.918034    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.918066 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.918054    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.918066 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.918064    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dab7faa-1a42-4139-9211-12bb1faf481c" containerName="cleanup"
Nov 01 21:00:21.922584 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.922564    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:21.926905 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.926879    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:00:21.927024 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.926994    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:00:21.927144 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.927127    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:00:21.927518 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.927493    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 21:00:21.929354 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.929331    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"]
Nov 01 21:00:21.983079 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.983040    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:21.983303 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.983088    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:21.983303 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:21.983214    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrr8\" (UniqueName: \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.084395 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.084355    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.084597 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.084406    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.084597 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.084449    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrr8\" (UniqueName: \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.085150 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.085123    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.086827 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.086805    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.092296 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.092272    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrr8\" (UniqueName: \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\") pod \"maas-api-revoke-sub-146ac48e-bmfvh\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.240725 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.240642    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:22.516140 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:22.516119    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"]
Nov 01 21:00:22.518330 ip-10-0-137-175 kubenswrapper[5041]: W1101 21:00:22.518301    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25795513_f62e_4f9a_ad5b_ee6f80a9f40f.slice/crio-77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe WatchSource:0}: Error finding container 77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe: Status 404 returned error can't find the container with id 77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe
Nov 01 21:00:23.165812 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:23.165706    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh" event={"ID":"25795513-f62e-4f9a-ad5b-ee6f80a9f40f","Type":"ContainerStarted","Data":"bec04a80966fe0108be86d36fc2c20f95ac84cd979fe48f9db6a67210bc15904"}
Nov 01 21:00:23.165812 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:23.165772    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh" event={"ID":"25795513-f62e-4f9a-ad5b-ee6f80a9f40f","Type":"ContainerStarted","Data":"77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe"}
Nov 01 21:00:23.182602 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:23.182555    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh" podStartSLOduration=2.182540505 podStartE2EDuration="2.182540505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:23.180262224 +0000 UTC m=+3710.901340901" watchObservedRunningTime="2026-11-01 21:00:23.182540505 +0000 UTC m=+3710.903619244"
Nov 01 21:00:24.174615 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:24.174583    5041 generic.go:356] "Generic (PLEG): container finished" podID="25795513-f62e-4f9a-ad5b-ee6f80a9f40f" containerID="bec04a80966fe0108be86d36fc2c20f95ac84cd979fe48f9db6a67210bc15904" exitCode=0
Nov 01 21:00:24.175090 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:24.174645    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh" event={"ID":"25795513-f62e-4f9a-ad5b-ee6f80a9f40f","Type":"ContainerDied","Data":"bec04a80966fe0108be86d36fc2c20f95ac84cd979fe48f9db6a67210bc15904"}
Nov 01 21:00:25.318381 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.318355    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:25.419679 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.419649    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\") pod \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") "
Nov 01 21:00:25.419870 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.419721    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\") pod \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") "
Nov 01 21:00:25.419870 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.419746    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\" (UniqueName: \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\") pod \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\" (UID: \"25795513-f62e-4f9a-ad5b-ee6f80a9f40f\") "
Nov 01 21:00:25.420125 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.420083    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca" pod "25795513-f62e-4f9a-ad5b-ee6f80a9f40f" (UID: "25795513-f62e-4f9a-ad5b-ee6f80a9f40f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:25.420264 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.420207    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:25.421915 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.421892    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca" pod "25795513-f62e-4f9a-ad5b-ee6f80a9f40f" (UID: "25795513-f62e-4f9a-ad5b-ee6f80a9f40f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:00:25.422010 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.421964    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8" pod "25795513-f62e-4f9a-ad5b-ee6f80a9f40f" (UID: "25795513-f62e-4f9a-ad5b-ee6f80a9f40f"). InnerVolumeSpecName "kube-api-access-gcrr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:25.521460 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.521366    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrr8\" (UniqueName: \"kubernetes.io/projected/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-kube-api-access-gcrr8\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:25.521460 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:25.521396    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25795513-f62e-4f9a-ad5b-ee6f80a9f40f-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:26.188085 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:26.188055    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh"
Nov 01 21:00:26.188085 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:26.188075    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146ac48e-bmfvh" event={"ID":"25795513-f62e-4f9a-ad5b-ee6f80a9f40f","Type":"ContainerDied","Data":"77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe"}
Nov 01 21:00:26.188326 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:26.188129    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77afa6aafcb96348c563a0d3b350ea7b51e11408410048f45e0b88a430f54ebe"
Nov 01 21:00:28.541423 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:00:28.541386    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 21:00:30.020409 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:30.020373    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"]
Nov 01 21:00:30.023106 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:30.023078    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4sh8v"]
Nov 01 21:00:30.427125 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:30.427094    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b4f20d-0020-4379-9935-54deb8995804" path="/var/lib/kubelet/pods/f5b4f20d-0020-4379-9935-54deb8995804/volumes"
Nov 01 21:00:32.116314 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.116276    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"]
Nov 01 21:00:32.116738 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.116729    5041 cpu_manager.go:414] "containerMap: removing container" podUID="25795513-f62e-4f9a-ad5b-ee6f80a9f40f" containerName="revoke-keys"
Nov 01 21:00:32.116813 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.116740    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25795513-f62e-4f9a-ad5b-ee6f80a9f40f" containerName="revoke-keys"
Nov 01 21:00:32.116866 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.116857    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="25795513-f62e-4f9a-ad5b-ee6f80a9f40f" containerName="revoke-keys"
Nov 01 21:00:32.121381 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.121364    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.125755 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.125727    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 21:00:32.125755 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.125727    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:00:32.126524 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.126490    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:00:32.126524 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.126490    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\""
Nov 01 21:00:32.128096 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.128068    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"]
Nov 01 21:00:32.190179 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.190147    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.190374 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.190202    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmgg\" (UniqueName: \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.190374 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.190303    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.291601 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.291561    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.291816 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.291621    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmgg\" (UniqueName: \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.291816 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.291655    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.292379 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.292359    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.294007 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.293986    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.299739 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.299712    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmgg\" (UniqueName: \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\") pod \"maas-api-revoke-sub-33f8b940-hdwqr\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.439245 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.439166    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:32.718948 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:32.718922    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"]
Nov 01 21:00:32.721127 ip-10-0-137-175 kubenswrapper[5041]: W1101 21:00:32.721096    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2d95f7_debe_4c09_9559_b23b86721b69.slice/crio-c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f WatchSource:0}: Error finding container c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f: Status 404 returned error can't find the container with id c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f
Nov 01 21:00:33.235247 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:33.235206    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr" event={"ID":"4c2d95f7-debe-4c09-9559-b23b86721b69","Type":"ContainerStarted","Data":"7ca5c9bef1f89e908c382cf5e73004eb7beeae94973452511385ff3017c11890"}
Nov 01 21:00:33.235846 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:33.235296    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr" event={"ID":"4c2d95f7-debe-4c09-9559-b23b86721b69","Type":"ContainerStarted","Data":"c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f"}
Nov 01 21:00:33.252160 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:33.252083    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr" podStartSLOduration=1.252064571 podStartE2EDuration="1.252064571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:00:33.250594404 +0000 UTC m=+3720.971673081" watchObservedRunningTime="2026-11-01 21:00:33.252064571 +0000 UTC m=+3720.973143249"
Nov 01 21:00:34.257804 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:34.254628    5041 generic.go:356] "Generic (PLEG): container finished" podID="4c2d95f7-debe-4c09-9559-b23b86721b69" containerID="7ca5c9bef1f89e908c382cf5e73004eb7beeae94973452511385ff3017c11890" exitCode=0
Nov 01 21:00:34.257804 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:34.254816    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr" event={"ID":"4c2d95f7-debe-4c09-9559-b23b86721b69","Type":"ContainerDied","Data":"7ca5c9bef1f89e908c382cf5e73004eb7beeae94973452511385ff3017c11890"}
Nov 01 21:00:35.218016 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.217963    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-qzbgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:00:35.218250 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.218036    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-qzbgz" podUID="9873f918-5b69-4ddd-b8f5-e27afea58726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:00:35.255364 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.254883    5041 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-p6vnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:00:35.255364 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.254953    5041 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-p6vnd" podUID="3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:00:35.635220 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.635193    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:35.732300 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.732262    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\") pod \"4c2d95f7-debe-4c09-9559-b23b86721b69\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") "
Nov 01 21:00:35.732561 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.732389    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\") pod \"4c2d95f7-debe-4c09-9559-b23b86721b69\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") "
Nov 01 21:00:35.732561 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.732432    5041 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\" (UniqueName: \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\") pod \"4c2d95f7-debe-4c09-9559-b23b86721b69\" (UID: \"4c2d95f7-debe-4c09-9559-b23b86721b69\") "
Nov 01 21:00:35.732760 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.732729    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca" pod "4c2d95f7-debe-4c09-9559-b23b86721b69" (UID: "4c2d95f7-debe-4c09-9559-b23b86721b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:00:35.734580 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.734546    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca" pod "4c2d95f7-debe-4c09-9559-b23b86721b69" (UID: "4c2d95f7-debe-4c09-9559-b23b86721b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:00:35.734679 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.734615    5041 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg" pod "4c2d95f7-debe-4c09-9559-b23b86721b69" (UID: "4c2d95f7-debe-4c09-9559-b23b86721b69"). InnerVolumeSpecName "kube-api-access-grmgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:00:35.833842 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.833806    5041 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmgg\" (UniqueName: \"kubernetes.io/projected/4c2d95f7-debe-4c09-9559-b23b86721b69-kube-api-access-grmgg\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:35.833842 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.833843    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-openshift-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:35.834059 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:35.833859    5041 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c2d95f7-debe-4c09-9559-b23b86721b69-maas-api-xks-ca\") on node \"ip-10-0-137-175.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:00:36.285625 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:36.285598    5041 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr"
Nov 01 21:00:36.285851 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:36.285628    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f8b940-hdwqr" event={"ID":"4c2d95f7-debe-4c09-9559-b23b86721b69","Type":"ContainerDied","Data":"c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f"}
Nov 01 21:00:36.285851 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:36.285678    5041 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c448a1e084c473c488994a39b9f4be55fad066357a477db4a26db8133dc5073f"
Nov 01 21:00:38.155522 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:38.155490    5041 scope.go:122] "RemoveContainer" containerID="3ee0aea7bcdc20aec2c495995a88165140a61b77c4fffc821ffcc101ff79045c"
Nov 01 21:00:38.165103 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:38.165078    5041 scope.go:122] "RemoveContainer" containerID="860d4cfeea49da215b522496d4b666f9f7bc8e537635c32870055b71f2ce3de5"
Nov 01 21:00:38.173881 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:38.173861    5041 scope.go:122] "RemoveContainer" containerID="df0bdec866edac8729833f03c9c5c0fd7a5c4f4528b373546b896487947b2388"
Nov 01 21:00:38.182100 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:38.182077    5041 scope.go:122] "RemoveContainer" containerID="878b174eb2636b038829f2052c2f017c644fabe5915019b9d3212e710916b75c"
Nov 01 21:00:50.132577 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:00:50.132489    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d8bc659cc-72rg2_40a08d27-42f8-48e2-8511-0032f24c6fd6/authorino/0.log"
Nov 01 21:00:58.541576 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:00:58.541535    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 21:01:01.031868 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:01.031831    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 21:01:01.986236 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:01.986202    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-2rd9z_97308fcb-f04e-43de-8e97-c3144625c0ef/manager/0.log"
Nov 01 21:01:02.328139 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:02.328108    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cj2xb_a902a934-ca05-4ef4-a015-35ca5504e645/manager/0.log"
Nov 01 21:01:03.783966 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:03.783927    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-146ac48e-bmfvh_25795513-f62e-4f9a-ad5b-ee6f80a9f40f/revoke-keys/0.log"
Nov 01 21:01:03.938975 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:03.938939    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-32889fd7-7tsnr_275f3e08-3d1b-4a09-bcb4-df9f9e561c83/revoke-keys/0.log"
Nov 01 21:01:04.106448 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.106415    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-33f8b940-hdwqr_4c2d95f7-debe-4c09-9559-b23b86721b69/revoke-keys/0.log"
Nov 01 21:01:04.281034 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.280999    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3d4156f1-p4tg8_84339091-243a-46a3-ab24-df9a957dfd64/revoke-keys/0.log"
Nov 01 21:01:04.443356 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.443272    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-51617705-slmht_081ad37a-5755-41ea-8b56-7b40597cd747/revoke-keys/0.log"
Nov 01 21:01:04.602649 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.602615    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84c616b6-rgvsd_7d0f9bda-5cee-4500-a0db-3f6698821391/revoke-keys/0.log"
Nov 01 21:01:04.763677 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.763593    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-h4562_8ed93b50-80aa-488a-ad06-fe93447b6065/revoke-keys/0.log"
Nov 01 21:01:04.935272 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:04.935242    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d8d0d18c-pp7w8_424832e3-d49c-4772-a266-d6f7d3a40f5b/revoke-keys/0.log"
Nov 01 21:01:05.105936 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:05.105898    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eefe487b-f5dg5_4fd92a18-1af7-4bc8-adbe-579f02c5c8c4/revoke-keys/0.log"
Nov 01 21:01:05.271968 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:05.271938    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ffb079ce-crjcf_88bb9e28-0415-4c76-b260-882316d18735/revoke-keys/0.log"
Nov 01 21:01:07.217770 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:07.217735    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d8bc659cc-72rg2_40a08d27-42f8-48e2-8511-0032f24c6fd6/authorino/0.log"
Nov 01 21:01:07.421397 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:07.421358    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/1.log"
Nov 01 21:01:07.592697 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:07.592667    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9hf26_64f36934-c791-4d4a-b443-81e23ecb97ac/manager/0.log"
Nov 01 21:01:07.764789 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:07.764749    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5fx26_74f17053-b4c9-40b0-8364-bbd0613232af/kuadrant-console-plugin/0.log"
Nov 01 21:01:08.423219 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:08.423185    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rrr99_09f298b9-1369-4358-8649-2d3e06b15ba0/manager/0.log"
Nov 01 21:01:08.965740 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:08.965704    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/0.log"
Nov 01 21:01:09.985577 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:09.985547    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-x5rr8_57934152-4f00-4fa3-af31-2d825cbae99f/kube-auth-proxy/0.log"
Nov 01 21:01:10.357448 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:10.357416    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-bnsdt_b1388a62-2172-466e-91f6-dcef2cf7525b/istio-proxy/0.log"
Nov 01 21:01:11.450908 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:11.450875    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-p6vnd_3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd/istio-proxy/0.log"
Nov 01 21:01:11.635544 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:11.635507    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-qzbgz_9873f918-5b69-4ddd-b8f5-e27afea58726/istio-proxy/0.log"
Nov 01 21:01:11.997650 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:11.997619    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv_5848925e-fc2c-43d4-b75f-a3b29a194244/istio-proxy/0.log"
Nov 01 21:01:12.490160 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:12.490127    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 21:01:13.591162 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:13.591126    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5cqq2_dc4fd9f8-3aea-42af-8483-acb9f8198223/main/0.log"
Nov 01 21:01:13.764953 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:13.764924    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-qvfwp_e79a81c6-16a9-4659-b063-05f6285aa2db/main/0.log"
Nov 01 21:01:14.113795 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:14.113753    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-94gsr_1cdc2c1c-0ee4-4b2e-83dc-29327158e7eb/main/0.log"
Nov 01 21:01:14.613755 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:14.613723    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7cp9c_1342869f-efd5-4140-bb37-c064736dfc96/main/0.log"
Nov 01 21:01:14.772482 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:14.772455    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-flhd5_5a81bba4-497d-453e-844a-a9e5cbc1f94b/main/0.log"
Nov 01 21:01:24.269274 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.269234    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2kn7n/must-gather-w2k54"]
Nov 01 21:01:24.271668 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.269700    5041 cpu_manager.go:414] "containerMap: removing container" podUID="4c2d95f7-debe-4c09-9559-b23b86721b69" containerName="revoke-keys"
Nov 01 21:01:24.271668 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.269714    5041 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2d95f7-debe-4c09-9559-b23b86721b69" containerName="revoke-keys"
Nov 01 21:01:24.271668 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.269854    5041 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2d95f7-debe-4c09-9559-b23b86721b69" containerName="revoke-keys"
Nov 01 21:01:24.273257 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.273239    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.279552 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.279529    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kn7n\"/\"kube-root-ca.crt\""
Nov 01 21:01:24.279695 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.279622    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2kn7n\"/\"default-dockercfg-kbd8m\""
Nov 01 21:01:24.279695 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.279692    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kn7n\"/\"openshift-service-ca.crt\""
Nov 01 21:01:24.282358 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.282334    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kn7n/must-gather-w2k54"]
Nov 01 21:01:24.337842 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.337811    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgm9\" (UniqueName: \"kubernetes.io/projected/3d669f9a-ce16-44d1-a894-430317b6fcc8-kube-api-access-9rgm9\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.338100 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.337971    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d669f9a-ce16-44d1-a894-430317b6fcc8-must-gather-output\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.439244 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.439210    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d669f9a-ce16-44d1-a894-430317b6fcc8-must-gather-output\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.439409 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.439277    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgm9\" (UniqueName: \"kubernetes.io/projected/3d669f9a-ce16-44d1-a894-430317b6fcc8-kube-api-access-9rgm9\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.439543 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.439523    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d669f9a-ce16-44d1-a894-430317b6fcc8-must-gather-output\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.447212 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.447178    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgm9\" (UniqueName: \"kubernetes.io/projected/3d669f9a-ce16-44d1-a894-430317b6fcc8-kube-api-access-9rgm9\") pod \"must-gather-w2k54\" (UID: \"3d669f9a-ce16-44d1-a894-430317b6fcc8\") " pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.590795 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.590746    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kn7n/must-gather-w2k54"
Nov 01 21:01:24.869655 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.869630    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kn7n/must-gather-w2k54"]
Nov 01 21:01:24.871483 ip-10-0-137-175 kubenswrapper[5041]: W1101 21:01:24.871457    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d669f9a_ce16_44d1_a894_430317b6fcc8.slice/crio-359e8b59cdc810ae9a80fa3d7c2abc24a793f85d3cd73e0f6376f94df776a4fd WatchSource:0}: Error finding container 359e8b59cdc810ae9a80fa3d7c2abc24a793f85d3cd73e0f6376f94df776a4fd: Status 404 returned error can't find the container with id 359e8b59cdc810ae9a80fa3d7c2abc24a793f85d3cd73e0f6376f94df776a4fd
Nov 01 21:01:24.873856 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:24.873839    5041 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:01:25.603940 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:25.603899    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kn7n/must-gather-w2k54" event={"ID":"3d669f9a-ce16-44d1-a894-430317b6fcc8","Type":"ContainerStarted","Data":"359e8b59cdc810ae9a80fa3d7c2abc24a793f85d3cd73e0f6376f94df776a4fd"}
Nov 01 21:01:27.756286 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:27.756248    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d549b_acc92340-bb89-4ff1-838e-08a6574531b9/global-pull-secret-syncer/0.log"
Nov 01 21:01:27.829820 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:27.829770    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5kdsg_6fb413b1-e547-411a-a27a-50716d2bd1ef/connection-checker/0.log"
Nov 01 21:01:27.852850 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:27.852827    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8n2gw_72e2c9d1-9ec9-4aa6-baed-78ca5facc4f4/connection-checker/0.log"
Nov 01 21:01:27.877093 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:27.877067    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ppbp7_087c078b-50e9-4942-ac02-6faf11ffecf3/connection-checker/0.log"
Nov 01 21:01:27.986370 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:27.986336    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qdn6f_a1c41cc7-e221-4bc2-aec2-af48980dc5eb/konnectivity-agent/0.log"
Nov 01 21:01:28.130585 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:28.130556    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-175.us-east-2.compute.internal_02aa65d6c5643e7fc09a137b81d3ea6b/haproxy/0.log"
Nov 01 21:01:28.550008 ip-10-0-137-175 kubenswrapper[5041]: E1101 21:01:28.549942    5041 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-922ms" containerName="extractor"
Nov 01 21:01:33.670441 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:33.670402    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kn7n/must-gather-w2k54" event={"ID":"3d669f9a-ce16-44d1-a894-430317b6fcc8","Type":"ContainerStarted","Data":"4bf9086c5bc849eb9ceca0eca920a2107fbbaaa977571d262879c1e7f2243b19"}
Nov 01 21:01:33.670441 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:33.670446    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kn7n/must-gather-w2k54" event={"ID":"3d669f9a-ce16-44d1-a894-430317b6fcc8","Type":"ContainerStarted","Data":"450fd804f384d8a1964d4588c195bb4f9b31c2d46adec062dedfd6107f3de396"}
Nov 01 21:01:33.685676 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:33.685625    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2kn7n/must-gather-w2k54" podStartSLOduration=1.930744018 podStartE2EDuration="9.685612721s" totalImagesPullingTime="7.754868703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:33.684627258 +0000 UTC m=+3781.405705934" watchObservedRunningTime="2026-11-01 21:01:33.685612721 +0000 UTC m=+3781.406691396"
Nov 01 21:01:33.765352 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:33.765320    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 21:01:34.210225 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.210183    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d8bc659cc-72rg2_40a08d27-42f8-48e2-8511-0032f24c6fd6/authorino/0.log"
Nov 01 21:01:34.655506 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.655475    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/1.log"
Nov 01 21:01:34.662402 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.662372    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/0.log"
Nov 01 21:01:34.692107 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.692068    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9hf26_64f36934-c791-4d4a-b443-81e23ecb97ac/manager/0.log"
Nov 01 21:01:34.720985 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.720960    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5fx26_74f17053-b4c9-40b0-8364-bbd0613232af/kuadrant-console-plugin/0.log"
Nov 01 21:01:34.966754 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:34.966667    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-2rd9z_97308fcb-f04e-43de-8e97-c3144625c0ef/manager/0.log"
Nov 01 21:01:35.156311 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:35.156275    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rrr99_09f298b9-1369-4358-8649-2d3e06b15ba0/manager/0.log"
Nov 01 21:01:36.770296 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:36.770258    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-x9tc5_a7cde0f8-1666-42a5-84e1-34f1ab468d1e/cluster-monitoring-operator/0.log"
Nov 01 21:01:37.021823 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.021701    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv2q4_5d3dfa71-36f6-448b-9339-750b874647e0/node-exporter/0.log"
Nov 01 21:01:37.053446 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.053397    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv2q4_5d3dfa71-36f6-448b-9339-750b874647e0/kube-rbac-proxy/0.log"
Nov 01 21:01:37.077133 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.077105    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fv2q4_5d3dfa71-36f6-448b-9339-750b874647e0/init-textfile/0.log"
Nov 01 21:01:37.250456 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.250413    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cj2xb_a902a934-ca05-4ef4-a015-35ca5504e645/manager/0.log"
Nov 01 21:01:37.437339 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.437308    5041 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"]
Nov 01 21:01:37.443039 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.443012    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.450263 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.450225    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"]
Nov 01 21:01:37.451514 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.451490    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"kube-root-ca.crt\""
Nov 01 21:01:37.474969 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.474934    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-proc\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.475125 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.474998    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-podres\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.475125 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.475096    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4wg\" (UniqueName: \"kubernetes.io/projected/a85602f1-dc49-43f5-bec0-43b5abe3d599-kube-api-access-wb4wg\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.475215 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.475136    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-sys\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.475303 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.475219    5041 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-lib-modules\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.513551 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.513520    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-nk7cf_d9752474-48e4-4bad-bc8a-205de8b30144/prometheus-operator-admission-webhook/0.log"
Nov 01 21:01:37.549054 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.549023    5041 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"openshift-service-ca.crt\""
Nov 01 21:01:37.549218 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.549161    5041 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z9gbm\"/\"default-dockercfg-fndqt\""
Nov 01 21:01:37.576697 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576668    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-podres\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.576932 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576716    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4wg\" (UniqueName: \"kubernetes.io/projected/a85602f1-dc49-43f5-bec0-43b5abe3d599-kube-api-access-wb4wg\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.576932 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576852    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-podres\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.576932 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576848    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-sys\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.576932 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576907    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-sys\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.576932 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.576924    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-lib-modules\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.577171 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.577044    5041 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-proc\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.577171 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.577051    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-lib-modules\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.577171 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.577151    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a85602f1-dc49-43f5-bec0-43b5abe3d599-proc\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.584276 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.584248    5041 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4wg\" (UniqueName: \"kubernetes.io/projected/a85602f1-dc49-43f5-bec0-43b5abe3d599-kube-api-access-wb4wg\") pod \"perf-node-gather-daemonset-sxs79\" (UID: \"a85602f1-dc49-43f5-bec0-43b5abe3d599\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:37.759095 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:37.759007    5041 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:38.047538 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.047452    5041 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"]
Nov 01 21:01:38.050631 ip-10-0-137-175 kubenswrapper[5041]: W1101 21:01:38.050598    5041 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda85602f1_dc49_43f5_bec0_43b5abe3d599.slice/crio-012a94da835fadd637698db4ce20ea302b74ff14e7cd41ea164d297bbe91645e WatchSource:0}: Error finding container 012a94da835fadd637698db4ce20ea302b74ff14e7cd41ea164d297bbe91645e: Status 404 returned error can't find the container with id 012a94da835fadd637698db4ce20ea302b74ff14e7cd41ea164d297bbe91645e
Nov 01 21:01:38.157972 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.157945    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 21:01:38.223743 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.223708    5041 scope.go:122] "RemoveContainer" containerID="d99818d4e5c934c1052f7529c3cb183c09c6662e6873f71c58ed7033c95fe581"
Nov 01 21:01:38.235454 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.235431    5041 scope.go:122] "RemoveContainer" containerID="481ef2bec2cc12b22390a9ae722b13c1b4bb186dd59f031cbc4acd0958f405fa"
Nov 01 21:01:38.244294 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.244274    5041 scope.go:122] "RemoveContainer" containerID="ff1a67b9438bc48c4a116b84a56a43abda8d3a5e395dace15d7ca80a1cb3d89a"
Nov 01 21:01:38.254042 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.254006    5041 scope.go:122] "RemoveContainer" containerID="d1f72d6c218f1a334fd9ff305ac080f59d867e8c35f33a8d0482e1feb4b55423"
Nov 01 21:01:38.709006 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.708924    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79" event={"ID":"a85602f1-dc49-43f5-bec0-43b5abe3d599","Type":"ContainerStarted","Data":"7daa891d6ddcd97d4074e4de828a3290ebefe8257f474ea1f4d1c531febb967f"}
Nov 01 21:01:38.709006 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.708995    5041 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79" event={"ID":"a85602f1-dc49-43f5-bec0-43b5abe3d599","Type":"ContainerStarted","Data":"012a94da835fadd637698db4ce20ea302b74ff14e7cd41ea164d297bbe91645e"}
Nov 01 21:01:38.709276 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.709029    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:38.725804 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.725745    5041 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79" podStartSLOduration=1.725731275 podStartE2EDuration="1.725731275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:38.724021784 +0000 UTC m=+3786.445100452" watchObservedRunningTime="2026-11-01 21:01:38.725731275 +0000 UTC m=+3786.446809950"
Nov 01 21:01:38.943370 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:38.943329    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-hr6lk_b2ebd17c-0816-4e50-b17b-bb1ca3633730/networking-console-plugin/0.log"
Nov 01 21:01:39.344408 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:39.344366    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-2rd9z_97308fcb-f04e-43de-8e97-c3144625c0ef/manager/0.log"
Nov 01 21:01:39.533894 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:39.533863    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/1.log"
Nov 01 21:01:39.550242 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:39.550209    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-pk8qq_053d72d0-c2a0-4e08-a7bf-4a6c1c7174d0/console-operator/2.log"
Nov 01 21:01:40.194728 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:40.194696    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-6bdrz_f631778e-0855-4a7f-8e2d-6c0385f9bbbe/download-server/0.log"
Nov 01 21:01:40.788276 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:40.788241    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-qrm2q_2b1aad29-2c51-448a-92cd-abc01fe696d1/volume-data-source-validator/0.log"
Nov 01 21:01:41.544719 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:41.544683    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cj2xb_a902a934-ca05-4ef4-a015-35ca5504e645/manager/0.log"
Nov 01 21:01:41.950856 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:41.950269    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6svs9_6533ed5f-4009-45be-8d6f-b90f1848d640/dns/0.log"
Nov 01 21:01:41.974019 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:41.973976    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6svs9_6533ed5f-4009-45be-8d6f-b90f1848d640/kube-rbac-proxy/0.log"
Nov 01 21:01:42.075950 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:42.075917    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q6p64_9666c411-ce1d-4592-82de-51b88b02ea47/dns-node-resolver/0.log"
Nov 01 21:01:42.425276 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:42.425245    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 21:01:42.695619 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:42.695516    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bb76fff67-jsvjg_03bbc2e4-cb60-4916-ad80-b38466c75efd/registry/0.log"
Nov 01 21:01:42.780117 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:42.780082    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szb2b_f3d00d01-ecba-4d49-aa5c-c9ab0e6e6867/node-ca/0.log"
Nov 01 21:01:43.620142 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:43.620108    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-2rd9z_97308fcb-f04e-43de-8e97-c3144625c0ef/manager/0.log"
Nov 01 21:01:44.497736 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.497703    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-x5rr8_57934152-4f00-4fa3-af31-2d825cbae99f/kube-auth-proxy/0.log"
Nov 01 21:01:44.570258 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.570224    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-bnsdt_b1388a62-2172-466e-91f6-dcef2cf7525b/istio-proxy/0.log"
Nov 01 21:01:44.740690 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.740635    5041 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-sxs79"
Nov 01 21:01:44.762528 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.762341    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-p6vnd_3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd/istio-proxy/0.log"
Nov 01 21:01:44.796271 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.796239    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-qzbgz_9873f918-5b69-4ddd-b8f5-e27afea58726/istio-proxy/0.log"
Nov 01 21:01:44.851529 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.851477    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv_5848925e-fc2c-43d4-b75f-a3b29a194244/istio-proxy/0.log"
Nov 01 21:01:44.938912 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:44.938880    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 21:01:45.887267 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:45.887240    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-cj2xb_a902a934-ca05-4ef4-a015-35ca5504e645/manager/0.log"
Nov 01 21:01:46.704939 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:46.704905    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74zbl_a5aab77d-3a73-46b8-ba03-03e96ebf16f0/serve-healthcheck-canary/0.log"
Nov 01 21:01:47.412762 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.412730    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4rbdb_adfbd1ed-3a02-4334-b370-328f2a511729/insights-operator/2.log"
Nov 01 21:01:47.431939 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.431900    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4rbdb_adfbd1ed-3a02-4334-b370-328f2a511729/insights-operator/1.log"
Nov 01 21:01:47.455234 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.455202    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-922ms_379fc616-e382-4250-8ec7-34217917ea30/kube-rbac-proxy/0.log"
Nov 01 21:01:47.480259 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.480230    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-922ms_379fc616-e382-4250-8ec7-34217917ea30/exporter/0.log"
Nov 01 21:01:47.506246 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.506218    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-922ms_379fc616-e382-4250-8ec7-34217917ea30/extractor/0.log"
Nov 01 21:01:47.719139 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.719058    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-x5rr8_57934152-4f00-4fa3-af31-2d825cbae99f/kube-auth-proxy/0.log"
Nov 01 21:01:47.760283 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.760252    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-c5hj4-6c5dr_d60f81ca-b7da-4b99-899c-f8ba5651ffa2/insights-gathering/0.log"
Nov 01 21:01:47.766751 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.766720    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-bnsdt_b1388a62-2172-466e-91f6-dcef2cf7525b/istio-proxy/0.log"
Nov 01 21:01:47.890021 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.889979    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-p6vnd_3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd/istio-proxy/0.log"
Nov 01 21:01:47.909731 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.909703    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-qzbgz_9873f918-5b69-4ddd-b8f5-e27afea58726/istio-proxy/0.log"
Nov 01 21:01:47.950918 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:47.950016    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv_5848925e-fc2c-43d4-b75f-a3b29a194244/istio-proxy/0.log"
Nov 01 21:01:48.003295 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.003220    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 21:01:48.272337 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.271702    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-x5rr8_57934152-4f00-4fa3-af31-2d825cbae99f/kube-auth-proxy/0.log"
Nov 01 21:01:48.318310 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.318277    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-bnsdt_b1388a62-2172-466e-91f6-dcef2cf7525b/istio-proxy/0.log"
Nov 01 21:01:48.463917 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.463879    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-p6vnd_3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd/istio-proxy/0.log"
Nov 01 21:01:48.484853 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.484817    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-qzbgz_9873f918-5b69-4ddd-b8f5-e27afea58726/istio-proxy/0.log"
Nov 01 21:01:48.532587 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.532496    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv_5848925e-fc2c-43d4-b75f-a3b29a194244/istio-proxy/0.log"
Nov 01 21:01:48.595249 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:48.595219    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 21:01:50.028193 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.028155    5041 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"]
Nov 01 21:01:50.031331 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.031301    5041 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffb079ce-crjcf"]
Nov 01 21:01:50.212385 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.212346    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d8bc659cc-72rg2_40a08d27-42f8-48e2-8511-0032f24c6fd6/authorino/0.log"
Nov 01 21:01:50.226311 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.226286    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/1.log"
Nov 01 21:01:50.232534 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.232511    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-msx7k_9d8742c6-60a8-4b53-b58e-ced648946d28/manager/0.log"
Nov 01 21:01:50.245234 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.245207    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-9hf26_64f36934-c791-4d4a-b443-81e23ecb97ac/manager/0.log"
Nov 01 21:01:50.258120 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.258100    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5fx26_74f17053-b4c9-40b0-8364-bbd0613232af/kuadrant-console-plugin/0.log"
Nov 01 21:01:50.430105 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.430016    5041 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bb9e28-0415-4c76-b260-882316d18735" path="/var/lib/kubelet/pods/88bb9e28-0415-4c76-b260-882316d18735/volumes"
Nov 01 21:01:50.550009 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.549980    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rrr99_09f298b9-1369-4358-8649-2d3e06b15ba0/manager/0.log"
Nov 01 21:01:50.778140 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.778065    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-nwc4n_2294303a-f8f8-439d-869a-a16d8fe60a1a/manager/0.log"
Nov 01 21:01:50.800635 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:50.800611    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-nwc4n_2294303a-f8f8-439d-869a-a16d8fe60a1a/copy-manifests/0.log"
Nov 01 21:01:51.562635 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.562599    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c66b786f-x5rr8_57934152-4f00-4fa3-af31-2d825cbae99f/kube-auth-proxy/0.log"
Nov 01 21:01:51.600275 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.600247    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-bnsdt_b1388a62-2172-466e-91f6-dcef2cf7525b/istio-proxy/0.log"
Nov 01 21:01:51.721521 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.721488    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-p6vnd_3662c1fb-5ab7-4dbd-b7d4-fd8d434c43cd/istio-proxy/0.log"
Nov 01 21:01:51.742278 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.742249    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-qzbgz_9873f918-5b69-4ddd-b8f5-e27afea58726/istio-proxy/0.log"
Nov 01 21:01:51.776405 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.776375    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-666dbc6559-984rv_5848925e-fc2c-43d4-b75f-a3b29a194244/istio-proxy/0.log"
Nov 01 21:01:51.827678 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:51.827578    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846f8d77f5-km598_c14186d7-dc3e-4df1-9358-806da5cbc684/router/0.log"
Nov 01 21:01:52.683356 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:52.683311    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-f9d557c9-9x2z8_95343c71-5cc5-4b8d-a194-6eaa3433eb6f/manager/0.log"
Nov 01 21:01:53.863725 ip-10-0-137-175 kubenswrapper[5041]: I1101 21:01:53.863689    5041 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f997f45fb-2rd9z_97308fcb-f04e-43de-8e97-c3144625c0ef/manager/0.log"